---
id: "20260815-0224-is-the-widrowhoff-lms"
title: "Is the Widrow–Hoff LMS rule established as a stochastic gradient descent method by a Tier 1–2 primary source, not just Wikipedia?"
type: "capture"
status: "promoted"
origin: "batch"
promoted_to: ["30-notes/claim-bottou-2010-classifies-widrow-hoff-lms-as-sgd-matching-original-algorithm.md","30-notes/claim-widrow-hoff-1960-original-paper-describes-lms-as-stochastic-steepest-descent.md","30-notes/claim-widrow-lehr-1990-lms-instantaneous-gradient-unbiased-estimate.md","40-entities/entity-ted-hoff.md","40-entities/entity-herbert-robbins.md","40-entities/entity-sutton-monro.md"]
not_promoted: ["Widrow & Stearns, Adaptive Signal Processing (1985), ref. [30] in the 1990 paper — no PDF or full-text excerpt locatable this session; left as a further lead on claim-widrow-lehr-1990-lms-instantaneous-gradient-unbiased-estimate.md rather than promoted, since nothing beyond the citation itself was read. Worth a library/archive.org pass in a future session.","The Robbins–Siegmund 1971 convergence theorem, cited by Bottou for SGD's convergence theory — a different, later Robbins-coauthored result from the 1951 Robbins–Monro paper already grounded in the vault. Not load-bearing for any claim here, so recorded only as a distinguishing note on the new entity-herbert-robbins.md page rather than promoted as its own claim-note."]
writer_model: "claude-sonnet-5"
date_created: "2026-08-15T00:00:00.000Z"
provenance: "batch run, 2026-08-15"
derived_from: []
tags: ["widrow","hoff","lms","adaline","sgd","stochastic-gradient-descent","bottou","stochastic-approximation","history-of-ml","sourcing-floor","backpropagation-origins"]
sources: [{"source_url":"https://leon.bottou.org/publications/pdf/compstat-2010.pdf","source_title":"Large-Scale Machine Learning with Stochastic Gradient Descent","source_author":"Léon Bottou","source_date":2010,"source_venue":"Author's own site (leon.bottou.org); presented at COMPSTAT'2010, 19th International Conference on Computational Statistics","source_tier":1,"source_sha":"c72c3bf5fe4b885b7719f49a770dcccfeb734473b1d6402da41bfeff5312257d","source_quote":"The stochastic gradient descent for the Perceptron, for the Adaline, and for k-Means match the algorithms proposed in the original papers.","source_delight":"A machine-learning researcher's own working paper puts the 1960 Widrow-Hoff update rule in the same table, and under the same name, as the stochastic gradient descent used to train today's large models.","source_note":"Fetched fresh this session via extract_pdf (TLS verified). Clean pdftotext extraction, no OCR noise. Both quoted passages verified grounded via quote_check against the extracted text."},{"source_url":"https://www-isl.stanford.edu/~widrow/papers/c1960adaptiveswitching.pdf","source_title":"Adaptive Switching Circuits","source_author":"Bernard Widrow and Marcian E. Hoff","source_date":1960,"source_venue":"1960 IRE WESCON Convention Record, Part 4, pp. 96-104","source_tier":1,"source_sha":"10c507b3f29c3a5162ad22fcc36e0eacfa7d28245c656ed5960de2b21ab150c1","source_quote":"is equivalent\nto. searching a parabolic stochastic surface (having\nas many dimensions as.there are a's) for a minimm,","source_note":"Fetched fresh this session via extract_pdf (TLS verified) from Widrow's own Stanford ISL page — the original venue, not a mirror. Extraction method was OCR (the scanned WESCON proceedings have no text layer), and OCR quality is visibly rough throughout (stray periods, 'minimm' for 'minimum', occasional dropped/garbled words). The quoted phrase above is transcribed exactly as extract_pdf rendered it and passed quote_check verbatim against that OCR text; the underlying typeset original almost certainly reads '...is equivalent to searching a parabolic stochastic surface...for a minimum.' Treat minor spelling artifacts as OCR noise, not authorial wording."},{"source_url":"https://isl.stanford.edu/~widrow/papers/j199030years.pdf","source_title":"30 Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation","source_author":"Bernard Widrow and Michael A. Lehr","source_date":"1990-09","source_venue":"Proceedings of the IEEE, vol. 78, no. 9, pp. 1415-1442, September 1990","source_tier":1,"source_sha":"f41e3cc5c1e8cc27e49dd78e69dfd823721caf49b944ede19bae5f5ccf8fdfd3","source_quote":"It i s shown in [30] that the instantaneous gradient i s an unbiased estimate of the true gradient.","source_note":"Fetched fresh this session via extract_pdf (TLS verified) from Widrow's own Stanford ISL page. Method pdftotext; text is largely clean but the two-column layout occasionally interleaves columns line-by-line and a stray space appears inside 'is' ('i s') throughout the document — an extraction artifact, not the source's own typography. Quote verified grounded via quote_check against the extracted text. Reference [30] in this quote is Widrow & Stearns, 'Adaptive Signal Processing' (Prentice-Hall, 1985) — see Further leads."}]
seek_code_commit: "17d9798"
---


This capture directly answers the open question raised in [[question-verify-lms-is-stochastic-gradient-descent-primary]], which flagged that [[claim-amari-stochastic-descent-and-widrow-hoff-lms-are-sgd-siblings]] rests its "Widrow–Hoff LMS is a stochastic gradient descent method" leg on a Tier-4 Wikipedia sentence alone — a technical-mechanism claim below the sourcing floor. Three Tier 1 primary sources were located and read directly this session; together they resolve the core question **yes**, though with a precise nuance about terminology recorded in the second and third claims below.

## Claim: A named researcher's own technical paper explicitly classifies the Widrow–Hoff Adaline/LMS update rule as an instance of stochastic gradient descent, stating that the classification matches the algorithm in Widrow and Hoff's original paper

**Claim type**: technical-mechanism (Tier 1–2 required; met — Léon Bottou's own paper, on his own venue).

Léon Bottou's 2010 paper "Large-Scale Machine Learning with Stochastic Gradient Descent" — hosted on his own site, leon.bottou.org — contains a table titled "Stochastic gradient algorithms for various learning systems" whose first row is headed "Adaline (Widrow and Hoff, 1960)," giving the per-sample weight update rule alongside entries for the Perceptron (Rosenblatt, 1957), k-Means, SVM, and Lasso. Bottou states directly: "The stochastic gradient descent for the Perceptron, for the Adaline, and for k-Means match the algorithms proposed in the original papers." This is a named, credentialed source's own technical analysis, on his own venue, explicitly and unambiguously placing the Widrow–Hoff LMS/Adaline rule inside the stochastic-gradient-descent family — not a paraphrase or aggregator summary. It directly answers the sourcing gap [[question-verify-lms-is-stochastic-gradient-descent-primary]] identified.

## Claim: Widrow and Hoff's own original 1960 paper already describes the LMS/Adaline error surface as a "stochastic" surface, searched one training pattern at a time by the method of steepest descent

**Claim type**: technical-mechanism / historical (Tier 1–2 required; met — the inventors' own original paper, on Widrow's own site, the original venue).

"Adaptive Switching Circuits" (Widrow & Hoff, 1960 IRE WESCON Convention Record) is the paper that introduced the LMS/Adaline rule. In its own words, adjusting the adaptive weights "is equivalent to searching a parabolic stochastic surface... for a minimum," and "the method of searching that has proven most useful is the method of steepest descent. Vector adjustment changes are made in the direction of the gradient." The paper further specifies that the adaptation procedure used "requires an extremely small sample size per iteration cycle, namely one pattern" — i.e., a single-sample (rather than full-batch) gradient step. This is the earliest primary description of the mechanism later named "stochastic gradient descent": the inventors themselves, in 1960, called the surface being searched "stochastic" and described searching it one sample at a time via the gradient. The paper does not use the compound phrase "stochastic gradient descent" (that phrase postdates 1960 usage in this literature), but the mechanism and the word "stochastic," applied to exactly this surface-search procedure, are both present in the founding primary document.

## Claim: Widrow's own 1990 retrospective (with Lehr) proves the mathematical property that defines stochastic gradient descent — that the per-sample "instantaneous gradient" used by LMS is an unbiased estimate of the true (expected) gradient — again without using the compound term "stochastic gradient descent" itself

**Claim type**: technical-mechanism (Tier 1–2 required; met — Widrow's own retrospective paper, on his own site).

In "30 Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation" (Widrow & Lehr, *Proceedings of the IEEE*, 1990), Widrow derives the μ-LMS algorithm as approximate steepest descent on the mean-square-error surface, using "an instantaneous gradient based upon the square of the instantaneous linear error" in place of the true gradient, "because it is readily available from a single data sample," noting that computing the true gradient "would involve averaging the instantaneous gradients associated with all patterns in the training set" and is "usually impractical." The paper states plainly: "It is shown in [30] that the instantaneous gradient is an unbiased estimate of the true gradient" (ref. [30] is Widrow & Stearns, *Adaptive Signal Processing*, Prentice-Hall, 1985). An update rule that substitutes a single-sample, unbiased estimate of the expected gradient for the true gradient, and iterates on that estimate, is stochastic gradient descent by mathematical definition — this is the property SGD is named for. Widrow himself proves it holds for LMS in this Tier 1 primary source, again without ever writing the compound phrase "stochastic gradient descent."

> [!note] Seek's commentary:
> The honest shape of the answer is: yes, but the word order matters. No Tier 1 Widrow paper found this session uses the literal phrase "stochastic gradient descent" for LMS — that specific compound term appears to be a later retrospective label (exactly the kind Bottou applies from outside, in 2010, fifty years after the fact). What Widrow's own 1960 and 1990 papers do establish, in their own words, is every piece of the mechanism the label names: a "stochastic" surface, single-sample ("one pattern per iteration") steepest descent, and a proof that the single-sample gradient is an unbiased estimator of the true gradient. Whether "established as an SGD method" requires the exact phrase or just the exact mechanism is a judgment call for whoever promotes this — I've recorded both the mechanism-level primary grounding and the terminology gap so that call can be made with full information, not papered over.

## Safety flags

None. All three sources fetched this session — Bottou's own site (leon.bottou.org) and Widrow's own Stanford ISL pages (isl.stanford.edu / www-isl.stanford.edu) — are straightforward academic PDFs: a working paper and two IEEE-society publications reproduced by their authors on their own institutional pages. No addressed-to-AI language, override language, claimed authority, tier self-assignment, file-system instructions, credential requests, or urgency framing on any of them. All three extract_pdf fetches reported `tls: "verified"`.

## Further leads

- Widrow & Stearns, *Adaptive Signal Processing* (Prentice-Hall, 1985) — reference [30] in the 1990 paper and the text most directly named in [[question-verify-lms-is-stochastic-gradient-descent-primary]]'s "what would answer it" list. No PDF or full-text excerpt was locatable via web search this session (only a Google Books listing and secondhand course-slide summaries); this is the standard textbook derivation and would be the strongest possible single source if obtained — worth a library/archive.org pass in a future session.
- Bottou's paper cites the Robbins–Siegmund theorem (Robbins and Siegmund, 1971) for SGD convergence under mild conditions — a *different*, later Robbins-coauthored convergence result from the 1951 Robbins–Monro paper already grounded in the vault at [[claim-robbins-monro-1951-stochastic-approximation]]; worth distinguishing the two if either is cited again.
- leon.bottou.org is a host not previously seen in the vault's other Widrow/LMS notes (isl.stanford.edu / www-isl.stanford.edu appear repeatedly); worth checking against the found-page host cap if this source is promoted.

## Entity candidates

- Herbert Robbins and Sutton Monro — person (pair) — the older, foundational figures the whole "SGD family" claim in this cluster ultimately rests on: Bottou's own 2010 paper explicitly invokes "the stochastic approximation literature" as the source of SGD's convergence theory, and the vault already grounds their 1951 paper at [[claim-robbins-monro-1951-stochastic-approximation]]. Flagged first, per the standing instruction to name the founding figure a priority/ancestry claim rests on before its co-authors — no standalone entity page found for either.
- Frank Rosenblatt — person — the contemporaneous 1950s figure Bottou's own table places directly alongside Adaline as a parallel SGD instance (Perceptron, 1957), and whom Widrow & Lehr's 1990 paper contrasts LMS against in dividing "error-correction rules" from "steepest-descent rules." No entity page found.
- Léon Bottou — person — author of the source that most directly and explicitly answers this capture's core question; a named, currently-active ML researcher (was at NEC Labs America at time of writing). No entity page found.
- Marcian "Ted" Hoff — person — co-inventor of the LMS/Adaline rule with Widrow in the 1960 primary source quoted above; the vault already has [[claim-ted-hoff-widrow-phd-student-architected-intel-4004]] but no standalone entity page was found for Hoff himself.
- Bernard Widrow — person — already has [[entity-bernard-widrow]]; author of two of this capture's three primary sources, thirty years apart (1960, 1990).