Discretize-then-optimize vs. optimize-then-discretize (DTO/OTD)

A standing distinction in numerical optimal control, predating and independent of neural networks: given a continuous-time dynamical system, one can either discretize the dynamics first and then differentiate the discrete recursion to get a gradient ("discretize-then-optimize," DTO), or derive the continuous-time adjoint/costate equations first and discretize those ("optimize-then-discretize," OTD). The two procedures are not guaranteed to commute, and in general do not.

Matters to this vault as the precise numerical-analysis name for the question underneath a cluster of history-of-backpropagation claims: differentiating a discretized neural-ODE forward pass (DTO) reproduces the Kelley-Bryson discrete costate recursion exactly, while solving Pontryagin's continuous adjoint ODE backward and then discretizing it (OTD) is the move modern neural-ODE papers make when they cite Pontryagin by name. claim-dto-and-otd-neural-ode-gradients-can-diverge shows these give provably different gradients, and can diverge outright for realistic nonlinearities; claim-ober-blobaum-junge-marsden-prove-discrete-adjoint-converges-to-continuous shows the two do converge under stated smoothness/coercivity/structure-preserving-discretization hypotheses. The synthesis sits at claim-kelley-bryson-pontryagin-costate-same-object-only-under-hypotheses.

References

• claim-dto-and-otd-neural-ode-gradients-can-diverge
• claim-ober-blobaum-junge-marsden-prove-discrete-adjoint-converges-to-continuous
• claim-kelley-bryson-pontryagin-costate-same-object-only-under-hypotheses
• claim-neural-odes-cite-pontryagin-adjoint-method-as-backprop — an OTD example
• claim-backprop-special-case-kelley-bryson-formula — a DTO example