Did Rumelhart, Hinton & Williams (1986) explicitly invoke random initial weights as a symmetry-breaking mechanism in the Nature paper or their own writings?
Short answer: yes, for the Nature paper itself. The letter contains one explicit, terse sentence stating exactly this. This was confirmed by fetching the primary document (a scan of Rumelhart, Hinton & Williams, "Learning representations by back-propagating errors," Nature 323, 9 October 1986, pp. 533–536) and reading its full OCR'd text directly, rather than relying on a secondary summary of it.
Claim: the Nature paper explicitly states that random initial weights serve to break symmetry
Claim type: Specific technical-mechanism claim (what a design choice in the algorithm is for). Requires Tier 1–2 per the sourcing floor.
Immediately after presenting the momentum-augmented gradient-descent update rule (their equation 9) and immediately before crediting independent co-discovery of the learning procedure, the paper's body text reads, in full: "To break symmetry we start with small random weights. Variants on the learning procedure have been discovered independently by David Parker (personal communication) and by Yann Le Cun." This is a single, standalone, declarative sentence — not an aside inside a larger explanatory passage. The paper does not, anywhere in its four pages, elaborate on why symmetric (e.g., equal or zero) initial weights would be a problem; the sentence simply asserts the fix without walking through the underlying reasoning (that reasoning — hidden units receiving identical error signals when weights start equal — is not present in this text as read).
Sourcing floor check: Cleared — Tier 1, the primary letter itself, read in full and quoted directly.
| Field | Value |
|---|---|
| source_url | https://gwern.net/doc/ai/nn/1986-rumelhart-2.pdf |
| source_author | David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams |
| source_date | 1986-10-09 |
| source_tier | 1 |
| exact_quote | "To break symmetry we start with small random weights." |
Claim: the paper's own worked example (the mirror-symmetry network) specifies its initial weights were random and drawn from a narrow uniform range
Claim type: Quantitative (specific numeric range) + technical-mechanism (how the demonstrated network was actually initialized). Requires Tier 1–2 per the sourcing floor.
The caption to Fig. 1 (the two-hidden-unit mirror-symmetry-detecting network) states: "The learning required 1,425 sweeps through the set of 64 possible input vectors, with the weights being adjusted on the basis of the accumulated gradient after each sweep. The values of the parameters in equation (9) were e = 0.1 and a = 0.9. The initial weights were random and were uniformly distributed between —0.3 and 0.3." This is the paper's own concrete instantiation of the general "small random weights" statement above, applied to one of its two demonstration networks.
Sourcing floor check: Cleared — Tier 1, the primary letter itself, read in full and quoted directly.
| Field | Value |
|---|---|
| source_url | https://gwern.net/doc/ai/nn/1986-rumelhart-2.pdf |
| source_author | David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams |
| source_date | 1986-10-09 |
| source_tier | 1 |
| exact_quote | "The initial weights were random and were uniformly distributed between —0.3 and 0.3." (Fig. 1 caption) |
Claim: Hinton has recounted, in his own words, that the symmetry-breaking rationale for random weights arose directly out of an objection he originally raised against backpropagation
Claim type: Historical/biographical claim about the internal discourse between Rumelhart and Hinton predating or surrounding the 1986 work. The rubric allows Tier 3–4 for uncontested historical claims but requires escalation to Tier 1–2 when the claim is "the load-bearing point of a note" — this is exactly that kind of claim here, since it bears on originality/motivation, not just settled trivia. Flagged accordingly.
According to a quotation attributed to Hinton and reproduced (with page citation) by an online computational-cognition course text, Hinton recalled of his initial reaction to Rumelhart's proposal of backpropagation: "I first of all explained to him why it wouldn't work, based on an argument in Rosenblatt's book, which showed essentially that it was an algorithm that couldn't break symmetry (...) The next argument I gave him was that it would get stuck in local minima (...)" The same course text separately states that Rumelhart's rejoinder was to propose random initial weights as the fix, though the "why can't we just break symmetry by having random initial weights?" phrasing was found only in general web-search synthesis text (not confirmed as a direct quotation with page number) and is not recorded here as a sourced claim for that reason.
This quotation is cited to Anderson, J.A. & Rosenfeld, E. (eds.), Talking Nets: An Oral History of Neural Networks (MIT Press, 2000), ch. 16, p. 376 — an oral-history interview transcript, which would ordinarily be a strong (Tier 1–2, first-person) source. However, the actual book could not be independently accessed (Internet Archive holds only a restricted loan copy with no search-inside function reachable here), so this capture is relying on a secondary reproduction of the quote rather than the book itself.
Sourcing floor check: NOT independently cleared at Tier 1–2 — recorded and flagged per the sourcing floor rule.
[unverified-mechanism -- needs primary: confirm exact wording and page against Anderson & Rosenfeld (2000), "Talking Nets," ch. 16, p. 376]
| Field | Value |
|---|---|
| source_url | https://com-cog-book.github.io/com-cog-book/features/multilayer-perceptron.html |
| source_author | Attributed to Geoffrey E. Hinton, via Anderson & Rosenfeld (2000), "Talking Nets," MIT Press |
| source_date | Talking Nets published 2000; interview date not given on the reproducing page |
| source_tier | 3 (secondary reproduction of a Tier-1-type primary quote; tier reflects what was actually accessed, not the underlying book) |
| exact_quote | "I first of all explained to him why it woulnd't work, based on an argument in Rosenblatt's book, which showed essentially that it was an algorithm that couldn't break symmetry (...) The next argument I gave him was that it would get stuck in local minima (...) Since you're bounded to get stuck in local minima, it wasn't really worth investigating (backpropagation)" [sic, typo "woulnd't" as it appears on the reproducing page] |
What remains unresolved
The research question asks about "the Nature paper or their own writings" (plural). Only the Nature letter itself was directly and fully verified above. The longer, more expository companion piece by the same three authors — "Learning Internal Representations by Error Propagation," in Rumelhart & McClelland (eds.), Parallel Distributed Processing, Vol. 1 (MIT Press, 1986), pp. 318–362, and its 1985 technical-report precursor (ICS-8506) — was not accessible in fetchable, text-extractable form (see access_failures above). Widely-repeated secondary paraphrases (e.g. general web-search synthesis, not independently quote-verified here) describe that longer chapter as containing a more elaborated explanation of why symmetric weights fail ("all hidden units connected directly to the output... will get identical error signals..."), but that specific wording was not confirmed against a primary or Tier 1–2 source in this session and is deliberately not recorded as a claim here.
Central question status: Confirmed for the Nature paper. Rumelhart, Hinton & Williams (1986), Nature 323, 533–536, contains the explicit sentence "To break symmetry we start with small random weights," directly invoking random weight initialization as a symmetry-breaking mechanism, plus a companion figure caption specifying the actual random-uniform range used in their worked example. Whether the same or an elaborated version of this claim also appears in the longer 1986 PDP book chapter by the same authors is [unverified -- could not confirm or deny after search], since that document could not be retrieved in readable form in this session.