Differentiating a discretized neural-ODE forward pass and discretizing the continuous adjoint ODE give provably different, sometimes divergent, gradients

neural-odesadjoint-methoddiscretize-then-optimizeoptimize-then-discretizenumerical-analysisbackpropagationkelley-brysonpontryagin

Training a neural-ODE layer admits two routes to a gradient. "Discretize-then-optimize" (DTO) differentiates the already-discretized forward recursion directly — for an explicit Euler step z1 = z0 + f(z0,θ), this reproduces the Kelley-Bryson/backprop chain-rule recursion exactly: ∂L/∂z1·(I + ∂f(z0,θ)/∂z0). "Optimize-then-discretize" (OTD) instead solves the continuous adjoint (costate) ODE backward in time and discretizes that — the move Chen et al. 2018 make, citing Pontryagin's adjoint method. Gholami, Keutzer & Biros (2019) prove these give literally different gradient expressions: OTD's adjoint recursion computes α1·(I + ∂f(z1,θ)/∂z1) — "the results from DTO approach (Eq. 9), and OTD (Eq. 10) can be completely different. This is because in general ∂f(z1,θ)/∂z1 ≠ ∂f(z0,θ)/∂z0."

The per-step discrepancy is formally small ("the error in OTD and DTO's gradient scales as O(dt)") but, in the paper's own words, kept verbatim rather than smoothed into an intuitive-sounding "large step" reading: "this error can become quite large for small time step sizes." Empirically, backward-in-time RK45 integration of the OTD adjoint "leads to divergent training in the first epoch" in the paper's SqueezeNext and ResNet-18 experiments on Cifar-10/100 (Figs. 3–5); the paper's finding for ReLU and non-ReLU activations and general convolution operators alike is its separate numerical-instability result for reversing the forward ODE (Figs. 1, 7), not the training-divergence demonstration. ANODE, the paper's own fix, is explicitly a DTO method — i.e., it recovers the Kelley-Bryson-style discrete recursion rather than the continuous Pontryagin costate ODE — specifically because DTO "provides unconditionally accurate gradients" where OTD does not.

See claim-kelley-bryson-pontryagin-costate-same-object-only-under-hypotheses for how this fits against the two notes it was retrieved to adjudicate.