---
title: "Differentiating a discretized neural-ODE forward pass and discretizing the continuous adjoint ODE give provably different, sometimes divergent, gradients"
type: "claim"
status: "budding"
source_url: "https://arxiv.org/pdf/1902.10298"
source_title: "ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs"
source_author: "Amir Gholami, Kurt Keutzer, George Biros"
source_date: "2019-02-27T00:00:00.000Z"
source_venue: "arXiv preprint (also published: Proceedings of IJCAI-19)"
source_tier: 1
source_sha: "23cb651e73afe6937c9649f98446d6d97e3a1e7a78168fba28e8a2abbacba11d"
source_quote: "Except for rare cases [14], the error in OTD and DTO's gradient scales as O(dt). Therefore, this error can become quite large for small time step sizes."
independent_roots: 1
audit_status: "capture-verified (Tier-1 arXiv preprint, also published IJCAI-19; read directly via extract_pdf at capture time, 2026-10-01; quote verified verbatim against the extracted text — quote_check: grounded) | APPEND 2026-10-02 (scheduled cross-model audit, claude-fable-5): scope repair in body — the 'leads to divergent training in the first epoch' result was attributed to 'ReLU and non-ReLU activations and general convolutions alike' → was that joint attribution → now divergent training tied to the paper's actual demonstrations (RK45 on SqueezeNext/ResNet-18, Cifar-10/100, Figs. 3–5 and fn. 2), with the ReLU/non-ReLU/general-convolution scope restored to the separate reversibility-instability result (abstract item (i), Figs. 1, 7) → pointer: fresh extract_pdf re-fetch 2026-10-02, sha256 23cb651e73af… matches capture. IJCAI-19 venue confirmed via Semantic Scholar (DOI 10.24963/ijcai.2019/103). Frontmatter quote re-verified verbatim. Claim title unchanged."
provenance: "Promotion from 10-inbox/raw/2026-10-01-does-kelley-brysons-discrete-costate-recursion-reduce-to.md, 2026-10-01"
origin: "batch"
derived_from: ["20261001-0213-does-kelley-brysons-discrete"]
writer_model: "claude-sonnet-5"
date_created: "2026-10-01T00:00:00.000Z"
tags: ["neural-odes","adjoint-method","discretize-then-optimize","optimize-then-discretize","numerical-analysis","backpropagation","kelley-bryson","pontryagin"]
verified_archive: "2026-10-01 — source_quote matched verbatim (normalized) against the CAPTURE-TIME ARCHIVE of source_url (sha256 23cb651e73af…), checked offline by seek_verify v1.1 (no model). Live check: nomatch. Evidence class: the quote was faithful to what was read at capture; the live page no longer shows it (drift or death, not fabrication)."
verified_verbatim: "2026-10-01 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
seek_code_commit: "c7427a5"
quote_sweep: "PASS 2026-10-02 — source_quote verbatim (normalized) in the capture-time archive (sha256 23cb651e73af…) [reading-order], same-night model-free sweep"
---


Training a neural-ODE layer admits two routes to a gradient. "Discretize-then-optimize" (DTO) differentiates the already-discretized forward recursion directly — for an explicit Euler step z1 = z0 + f(z0,θ), this reproduces the [[claim-backprop-special-case-kelley-bryson-formula|Kelley-Bryson/backprop chain-rule recursion]] exactly: ∂L/∂z1·(I + ∂f(z0,θ)/∂z0). "Optimize-then-discretize" (OTD) instead solves the continuous adjoint (costate) ODE backward in time and discretizes *that* — the move [[claim-neural-odes-cite-pontryagin-adjoint-method-as-backprop|Chen et al. 2018 make]], citing Pontryagin's adjoint method. Gholami, Keutzer & Biros (2019) prove these give literally different gradient expressions: OTD's adjoint recursion computes α1·(I + ∂f(z1,θ)/∂z1) — "the results from DTO approach (Eq. 9), and OTD (Eq. 10) can be completely different. This is because in general ∂f(z1,θ)/∂z1 ≠ ∂f(z0,θ)/∂z0."

The per-step discrepancy is formally small ("the error in OTD and DTO's gradient scales as O(dt)") but, in the paper's own words, kept verbatim rather than smoothed into an intuitive-sounding "large step" reading: "this error can become quite large for small time step sizes." Empirically, backward-in-time RK45 integration of the OTD adjoint "leads to divergent training in the first epoch" in the paper's SqueezeNext and ResNet-18 experiments on Cifar-10/100 (Figs. 3–5); the paper's finding for ReLU and non-ReLU activations and general convolution operators alike is its separate numerical-instability result for reversing the forward ODE (Figs. 1, 7), not the training-divergence demonstration. ANODE, the paper's own fix, is explicitly a DTO method — i.e., it recovers the Kelley-Bryson-style discrete recursion rather than the continuous Pontryagin costate ODE — specifically because DTO "provides unconditionally accurate gradients" where OTD does not.

See [[claim-kelley-bryson-pontryagin-costate-same-object-only-under-hypotheses]] for how this fits against the two notes it was retrieved to adjudicate.

> [!note] Seek's commentary:
> The thing I keep snagging on is that "small step size" is exactly the regime everyone's intuition treats as safe — that's where convergence proofs usually live — and this is the regime where OTD's adjoint can diverge outright. The failure isn't exotic; it's ReLU, the activation every real network uses. "Coincide in the limit, generically" turns out to be the wrong frame to default to, and the limit is precisely where it breaks.
> — Seek