Mandt, Hoffman and Blei (2017) model constant-learning-rate SGD near a loss minimum as an Ornstein–Uhlenbeck process

In "Stochastic Gradient Descent as Approximate Bayesian Inference" (JMLR 2017), Mandt, Hoffman and Blei observe that "Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution." Rather than converging to a point, constant-rate SGD keeps bouncing around a loss minimum, and the distribution of where it lands settles into a fixed stationary form. Their analysis approximates this Markov chain by a continuous-time Ornstein–Uhlenbeck (OU) process: near a minimum the loss is locally quadratic, so the gradient supplies a linear restoring force toward the bottom of the well while the minibatch noise plays the role of the driving randomness. SGD near a minimum is, on this account, OU mean-reversion in a quadratic basin.

This closes a bridge onto the vault's ML-optimization cluster. SGD's own lineage runs through stochastic approximation (claim-robbins-monro-1951-stochastic-approximation) and the LMS/stochastic-descent siblings (claim-amari-stochastic-descent-and-widrow-hoff-lms-are-sgd-siblings); its failure modes and geometry appear in claim-vanishing-gradient-chain-rule-pathology and claim-amari-1998-natural-gradient-fisher-steepest-descent, catalogued under moc-backpropagation-origins and backpropagation-gap. The same OU restoring-force object governs fossil stasis (claim-ou-model-recasts-stasis-as-active-mean-reversion-to-an-optimum) and interest rates (claim-ornstein-uhlenbeck-process-links-brownian-motion-and-the-vasicek-model); the three legs are joined in observation-mean-reversion-to-an-optimum-recurs-across-fossil-stasis-bonds-and-sgd.

The OU description is an approximation resting on stated assumptions — a constant learning rate, a locally quadratic loss, and roughly Gaussian gradient noise. The capture flagged this as the load-bearing mechanism of its whole cross-domain bridge and advised double-checking before publication; the note therefore stays seedling, with the scope and assumptions of the approximation routed to question-verify-mandt-2017-constant-sgd-ornstein-uhlenbeck.