---
id: "20260705-0209-did-rumelhart-hinton-williams"
title: "Did Rumelhart, Hinton & Williams (1986) explicitly invoke random initial weights as a symmetry-breaking mechanism in the Nature paper or their own writings?"
type: "capture"
status: "promoted"
promoted_to: ["30-notes/claim-rhw-1986-demonstration-not-invention.md (dedupe: the symmetry-breaking sentence was already there citing this capture; Fig-1 initialization detail (uniform −0.3..0.3, 1,425 sweeps) and 'asserts the fix without the why' folded in as an addendum)"]
not_promoted: ["Hinton Talking-Nets objection story (couldn't-break-symmetry via Rosenblatt) — Tier 3 secondary of a restricted-loan primary; verified NOT in the vault's cached Werbos-adjacent Talking Nets excerpts; getting ch. 16 p. 376 is a Cali-side library op, flag preserved in capture","PDP Vol. 1 chapter elaboration — DTIC ICS-8506 still 429-rate-limited at promotion time (retried); [unverified] stands"]
promotion_note: "Queen cycle 19, 2026-07-07."
origin: "batch"
model: "claude-sonnet-5"
date_created: "2026-07-05T00:00:00.000Z"
provenance: "batch run 2026-07-05; web research via WebSearch + WebFetch, plus direct full-text read (via the Read tool against a fetched PDF) of the primary Nature 1986 letter; harvested from 2026-06-29-what-did-the-1986-rumelhart-hinton-williams-nature-paper-add-that-earlier-formulations-did-not"
derived_from: []
tags: ["backpropagation","rumelhart-hinton-williams-1986","symmetry-breaking","weight-initialization","history-of-ai","primary-source-verification"]
primary_sources_consulted: [{"url":"https://gwern.net/doc/ai/nn/1986-rumelhart-2.pdf","author":"David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams","date":"1986-10-09","tier":1,"note":"Direct scan/mirror of the actual Nature paper 'Learning representations by back-propagating errors,' Nature 323, pp. 533-536 (received 1 May, accepted 31 July 1986). WebFetch's own summarization pass could not extract text from the JBIG2-compressed scan, but the Read tool, applied directly to the saved PDF binary, returned the full OCR'd body text of the letter. Quoted verbatim below."},{"url":"https://com-cog-book.github.io/com-cog-book/features/multilayer-perceptron.html","author":"com-cog-book.github.io (an online computational cognitive science course text); quoting Anderson, J.A. & Rosenfeld, E. (2000), 'Talking Nets: An Oral History of Neural Networks,' MIT Press, ch. 16 ('Geoffrey E. Hinton'), p. 376","date":"retrieved 2026-07-05","tier":3,"note":"Secondary reproduction of a Hinton oral-history quote. The original book (Talking Nets) could not be accessed directly — Internet Archive holds it only as a restricted/non-full-text-searchable loan item, so the quote below is recorded via this secondary transcription and flagged accordingly, not treated as independently Tier 1-2."},{"url":"https://chsasank.com/classic_papers/learning-representations-back-propogating-errors.html","author":"Sasank Chilamkurthy (chsasank.com, personal technical blog)","date":"retrieved 2026-07-05","tier":3,"note":"Used only as corroboration that its verbatim reproduction of the same passage matched the primary text read directly below; not relied on as the evidentiary source for any claim in this capture."}]
access_failures: [{"url":"https://apps.dtic.mil/sti/tr/pdf/ADA164453.pdf","reason":"Repeated HTTP 429 (rate-limited) on WebFetch across multiple attempts. This is the DTIC-hosted ICS-8506 technical report (Sept. 1985), the direct precursor to the longer PDP-book chapter version of this work; could not be checked for an independent occurrence of the symmetry-breaking sentence."},{"url":"https://www.nature.com/articles/323533a0","reason":"Redirects to an authentication/paywall gate (idp.nature.com); publisher's own hosted full text not accessible without institutional login."},{"url":"https://www.academia.edu/2520405/Learning_representations_by_back_propagating_errors","reason":"HTTP 403 Forbidden on WebFetch."},{"url":"PDP Vol. 1 book chapter (Rumelhart, Hinton & Williams, 'Learning Internal Representations by Error Propagation,' in Rumelhart & McClelland, eds., Parallel Distributed Processing, Vol. 1, MIT Press, 1986, pp. 318-362)","reason":"Not located in a fetchable, text-extractable form; only citation records and paywalled book-preview pages were found. This is the 'their own writings' half of the research question that remains unchecked directly — see commentary."}]
---


**Short answer: yes, for the Nature paper itself.** The letter contains one explicit, terse sentence stating exactly this. This was confirmed by fetching the primary document (a scan of Rumelhart, Hinton & Williams, "Learning representations by back-propagating errors," *Nature* 323, 9 October 1986, pp. 533–536) and reading its full OCR'd text directly, rather than relying on a secondary summary of it.

---

## Claim: the Nature paper explicitly states that random initial weights serve to break symmetry

**Claim type:** Specific technical-mechanism claim (what a design choice in the algorithm is *for*). Requires Tier 1–2 per the sourcing floor.

Immediately after presenting the momentum-augmented gradient-descent update rule (their equation 9) and immediately before crediting independent co-discovery of the learning procedure, the paper's body text reads, in full: "To break symmetry we start with small random weights. Variants on the learning procedure have been discovered independently by David Parker (personal communication) and by Yann Le Cun." This is a single, standalone, declarative sentence — not an aside inside a larger explanatory passage. The paper does not, anywhere in its four pages, elaborate on *why* symmetric (e.g., equal or zero) initial weights would be a problem; the sentence simply asserts the fix without walking through the underlying reasoning (that reasoning — hidden units receiving identical error signals when weights start equal — is not present in this text as read).

**Sourcing floor check:** Cleared — Tier 1, the primary letter itself, read in full and quoted directly.

| Field | Value |
|---|---|
| source_url | https://gwern.net/doc/ai/nn/1986-rumelhart-2.pdf |
| source_author | David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams |
| source_date | 1986-10-09 |
| source_tier | 1 |
| exact_quote | "To break symmetry we start with small random weights." |

---

## Claim: the paper's own worked example (the mirror-symmetry network) specifies its initial weights were random and drawn from a narrow uniform range

**Claim type:** Quantitative (specific numeric range) + technical-mechanism (how the demonstrated network was actually initialized). Requires Tier 1–2 per the sourcing floor.

The caption to Fig. 1 (the two-hidden-unit mirror-symmetry-detecting network) states: "The learning required 1,425 sweeps through the set of 64 possible input vectors, with the weights being adjusted on the basis of the accumulated gradient after each sweep. The values of the parameters in equation (9) were e = 0.1 and a = 0.9. The initial weights were random and were uniformly distributed between —0.3 and 0.3." This is the paper's own concrete instantiation of the general "small random weights" statement above, applied to one of its two demonstration networks.

**Sourcing floor check:** Cleared — Tier 1, the primary letter itself, read in full and quoted directly.

| Field | Value |
|---|---|
| source_url | https://gwern.net/doc/ai/nn/1986-rumelhart-2.pdf |
| source_author | David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams |
| source_date | 1986-10-09 |
| source_tier | 1 |
| exact_quote | "The initial weights were random and were uniformly distributed between —0.3 and 0.3." (Fig. 1 caption) |

---

## Claim: Hinton has recounted, in his own words, that the symmetry-breaking rationale for random weights arose directly out of an objection he originally raised against backpropagation

**Claim type:** Historical/biographical claim about the internal discourse between Rumelhart and Hinton predating or surrounding the 1986 work. The rubric allows Tier 3–4 for uncontested historical claims but requires escalation to Tier 1–2 when the claim is "the load-bearing point of a note" — this is exactly that kind of claim here, since it bears on originality/motivation, not just settled trivia. **Flagged accordingly.**

According to a quotation attributed to Hinton and reproduced (with page citation) by an online computational-cognition course text, Hinton recalled of his initial reaction to Rumelhart's proposal of backpropagation: "I first of all explained to him why it wouldn't work, based on an argument in Rosenblatt's book, which showed essentially that it was an algorithm that couldn't break symmetry (...) The next argument I gave him was that it would get stuck in local minima (...)" The same course text separately states that Rumelhart's rejoinder was to propose random initial weights as the fix, though the "why can't we just break symmetry by having random initial weights?" phrasing was found only in general web-search synthesis text (not confirmed as a direct quotation with page number) and is **not** recorded here as a sourced claim for that reason.

This quotation is cited to Anderson, J.A. & Rosenfeld, E. (eds.), *Talking Nets: An Oral History of Neural Networks* (MIT Press, 2000), ch. 16, p. 376 — an oral-history interview transcript, which would ordinarily be a strong (Tier 1–2, first-person) source. However, the actual book could not be independently accessed (Internet Archive holds only a restricted loan copy with no search-inside function reachable here), so this capture is relying on a secondary reproduction of the quote rather than the book itself.

**Sourcing floor check:** NOT independently cleared at Tier 1–2 — recorded and flagged per the sourcing floor rule.

`[unverified-mechanism -- needs primary: confirm exact wording and page against Anderson & Rosenfeld (2000), "Talking Nets," ch. 16, p. 376]`

| Field | Value |
|---|---|
| source_url | https://com-cog-book.github.io/com-cog-book/features/multilayer-perceptron.html |
| source_author | Attributed to Geoffrey E. Hinton, via Anderson & Rosenfeld (2000), "Talking Nets," MIT Press |
| source_date | Talking Nets published 2000; interview date not given on the reproducing page |
| source_tier | 3 (secondary reproduction of a Tier-1-type primary quote; tier reflects what was actually accessed, not the underlying book) |
| exact_quote | "I first of all explained to him why it woulnd't work, based on an argument in Rosenblatt's book, which showed essentially that it was an algorithm that couldn't break symmetry (...) The next argument I gave him was that it would get stuck in local minima (...) Since you're bounded to get stuck in local minima, it wasn't really worth investigating (backpropagation)" [sic, typo "woulnd't" as it appears on the reproducing page] |

---

## What remains unresolved

The research question asks about "the Nature paper **or** their own writings" (plural). Only the Nature letter itself was directly and fully verified above. The longer, more expository companion piece by the same three authors — "Learning Internal Representations by Error Propagation," in Rumelhart & McClelland (eds.), *Parallel Distributed Processing*, Vol. 1 (MIT Press, 1986), pp. 318–362, and its 1985 technical-report precursor (ICS-8506) — was not accessible in fetchable, text-extractable form (see `access_failures` above). Widely-repeated secondary paraphrases (e.g. general web-search synthesis, not independently quote-verified here) describe that longer chapter as containing a more elaborated explanation of *why* symmetric weights fail ("all hidden units connected directly to the output... will get identical error signals..."), but that specific wording was not confirmed against a primary or Tier 1–2 source in this session and is deliberately **not** recorded as a claim here.

> [!note] Seek's commentary:
> The main finding here is procedural as much as substantive: a WebFetch summarization pass on the raw Nature PDF twice reported the document as unreadable ("JBIG2-compressed," "no extractable text"). Reading the identical saved PDF binary directly with the Read tool succeeded completely and pulled clean OCR text for all four pages, including the exact sentence in question. The lesson repeats one already logged in the 2026-07-04 capture on this same paper: for quantitative/mechanism claims on a primary technical document, read the fetched artifact directly rather than trusting a fetch tool's own summary of it — the two can disagree sharply, and only one of them is checking the actual page.

**Central question status:** Confirmed for the Nature paper. Rumelhart, Hinton & Williams (1986), *Nature* 323, 533–536, contains the explicit sentence "To break symmetry we start with small random weights," directly invoking random weight initialization as a symmetry-breaking mechanism, plus a companion figure caption specifying the actual random-uniform range used in their worked example. Whether the same or an elaborated version of this claim also appears in the longer 1986 PDP book chapter by the same authors is `[unverified -- could not confirm or deny after search]`, since that document could not be retrieved in readable form in this session.