---
title: "Mandt, Hoffman and Blei (2017) model constant-learning-rate SGD near a loss minimum as an Ornstein–Uhlenbeck process"
type: "claim"
status: "seedling"
source_url: "https://arxiv.org/abs/1704.04289"
source_title: "Stochastic Gradient Descent as Approximate Bayesian Inference"
source_author: "Stephan Mandt, Matthew D. Hoffman, David M. Blei"
source_date: 2017
source_venue: "Journal of Machine Learning Research 18 (2017); 'Stochastic Gradient Descent as Approximate Bayesian Inference'"
source_quote: "Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution."
source_tier: 1
audit_status: "capture-verified (arXiv 1704.04289 referenced at capture time; queen independent re-check blocked in headless promotion. The capture itself flagged this as the load-bearing 'SGD-as-OU' mechanism and advised double-checking before publication; the OU claim is an approximation under stated assumptions — see [[question-verify-mandt-2017-constant-sgd-ornstein-uhlenbeck]]). Kept seedling."
provenance: "Promotion from 10-inbox/raw/2026-07-11-hop-stasis-is-an-ou-process.md, 2026-07-12"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-11-hop-stasis-is-an-ou-process.md"
writer_model: "claude-opus-4-8"
date_created: "2026-07-12T00:00:00.000Z"
tags: ["machine-learning","stochastic-gradient-descent","ornstein-uhlenbeck","optimization","loss-landscape","mean-reversion","approximate-bayesian-inference"]
---


In "Stochastic Gradient Descent as Approximate Bayesian Inference" (JMLR 2017), Mandt, Hoffman and Blei observe that "Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution." Rather than converging to a point, constant-rate SGD keeps bouncing around a loss minimum, and the distribution of where it lands settles into a fixed stationary form. Their analysis approximates this Markov chain by a continuous-time Ornstein–Uhlenbeck (OU) process: near a minimum the loss is locally quadratic, so the gradient supplies a linear restoring force toward the bottom of the well while the minibatch noise plays the role of the driving randomness. SGD near a minimum is, on this account, OU mean-reversion in a quadratic basin.

This closes a bridge onto the vault's ML-optimization cluster. SGD's own lineage runs through stochastic approximation ([[claim-robbins-monro-1951-stochastic-approximation]]) and the LMS/stochastic-descent siblings ([[claim-amari-stochastic-descent-and-widrow-hoff-lms-are-sgd-siblings]]); its failure modes and geometry appear in [[claim-vanishing-gradient-chain-rule-pathology]] and [[claim-amari-1998-natural-gradient-fisher-steepest-descent]], catalogued under [[moc-backpropagation-origins]] and [[backpropagation-gap]]. The same OU restoring-force object governs fossil stasis ([[claim-ou-model-recasts-stasis-as-active-mean-reversion-to-an-optimum]]) and interest rates ([[claim-ornstein-uhlenbeck-process-links-brownian-motion-and-the-vasicek-model]]); the three legs are joined in [[observation-mean-reversion-to-an-optimum-recurs-across-fossil-stasis-bonds-and-sgd]].

The OU description is an *approximation* resting on stated assumptions — a constant learning rate, a locally quadratic loss, and roughly Gaussian gradient noise. The capture flagged this as the load-bearing mechanism of its whole cross-domain bridge and advised double-checking before publication; the note therefore stays `seedling`, with the scope and assumptions of the approximation routed to [[question-verify-mandt-2017-constant-sgd-ornstein-uhlenbeck]].