A twin-prime number theorist's routine sanity check exposed a $475M hardware rounding defect in the Pentium

In June 1994, Lynchburg College mathematician Thomas Nicely was extending his estimate of Brun's constant (the sum of reciprocals of twin primes) onto a new Pentium machine when cross-checks against non-Pentium results diverged. As Vaughan Pratt's academic post-mortem describes it: "The error was first noticed within Intel and independently by Prof. Thomas Nicely... in the course of an ongoing project to estimate the sum of the reciprocals of the twin primes... Nicely publicized his discovery at the end of October, and it quickly became a cause célèbre on the Internet." (Tier 1, Pratt 1995)

The mechanism: "The Intel Pentium microprocessor makes occasional errors in floating point divisions due to five missing entries from a lookup table of quotient digits" — a lithography defect in the SRT division algorithm's programmable logic array. (Tier 1, Pratt 1995)

Intel initially resisted a full recall, then relented. Its own investor filing states plainly: "fourth quarter gross margin was reduced by $475 million." (Tier 1, Intel SEC filing, EX-99.2, Jan 1995)

Why this was hop-worthy

Linnainmaa's thesis formalized how to trace accumulated floating-point error back to its N local sources mathematically. This chain found the opposite direction playing out in the wild — a single missing hardware entry, surfaced not by an engineer testing the chip but by a number theorist doing unrelated pure math, cascading into the costliest arithmetic mistake in chip history.

Further leads

• Tim Coe (Vitesse Semiconductor) reverse-engineered the exact defective lookup-table geometry — a hardware-detective hook not followed here.
• The Mathisen–Moler software workaround (scaling operands to dodge the five bad cells) — a specific mitigation mechanism, not yet explored.

Hop chain

Seed: claim-linnainmaa-rounding-error-problem.md — Linnainmaa's 1970 thesis on the sensitivity of accumulated rounding error to local errors. Redirected away from AD/backprop per session instructions.

Hop 1: Patriot Missile software failure, Dhahran, Feb 25 1991

• Source: GAO Report, "Patriot Missile Defense: Software Problem Led to System Failure at Dhahran, Saudi Arabia" — https://cs.nyu.edu/~exact/resource/mirror/patriot.htm
• Hook type: Surprising claim / cross-domain bridge
• Hook: the same category of error the seed treats as abstract Taylor-coefficient math (accumulated truncation) killed 28 soldiers when a 24-bit register chopped 1/10's binary expansion, drifting the system clock ~0.34s over 100 hours of continuous operation.
• Why followed: it's the starkest possible "connect but extend" move — pure numerical-analysis theory into real, fatal consequence.
• Key findings: the error was a design mismatch (mobile reboot cadence assumed) more than an arithmetic one — the Patriot was never rebooted during the 100+ hour stationary deployment that let the drift accumulate.

Hop 2: William Kahan and the IEEE 754 standard

• Source: UC Berkeley Math Dept news — https://math.berkeley.edu/news/congratulations-professor-william-velvel-kahan ; ETHW Milestone — https://ethw.org/Milestones:IEEE_Standard_754_for_Binary_Floating-Point_Arithmetic,_1985
• Hook type: The person behind the thing (zoom out)
• Hook: Kahan, hired by Intel in 1976 to spec the 8087 coprocessor's arithmetic, turned that corporate spec into the global IEEE 754 standard governing how all rounding happens today — the standard the Patriot and Pentium both operated under (or around).
• Why followed: zoom-out from a specific disaster to the singular human who wrote the rules of rounding itself; won 1989 Turing Award for it.
• Key findings: IEEE 754 (1985) grew directly out of Intel's need for a defensible in-hardware arithmetic spec, not from an abstract standards committee.

Hop 3: The Pentium FDIV bug, 1994

• Source: Vaughan Pratt, "Anatomy of the Pentium Bug," Stanford CS, 1995 — http://boole.stanford.edu/pub/anapent.pdf
• Hook type: Mechanism question (zoom in)
• Hook: five missing entries in a 2,048-cell division lookup table, caused by a lithography error, produced wrong quotients in ~1 in 9 billion double-precision divisions.
• Why followed: same field (Kahan's arithmetic standard) made concrete as the most famous hardware rounding bug in history.
• Key findings: error rate was wildly non-uniform — "bruised" small integers near threshold values could error every few hundred divisions, far above the naive random-operand estimate.

Hop 4: Thomas Nicely's discovery

• Source: same Pratt 1995 paper; Intel SEC filing EX-99.2, Jan 17 1995 — https://www.intc.com/filings-reports/all-sec-filings/content/0000050863-95-000001/EX-99.txt
• Hook type: The person behind the thing / unfamiliar name (zoom out)
• Hook: an unrelated number-theory project (twin-prime reciprocal sums) is what actually surfaced the bug publicly, ahead of Intel's own internal QA catching it in the wild.
• Why followed: closes the loop on "who noticed" — completing the person-behind-the-thing arc the chain opened with Kahan.
• Key findings: Nicely reported to Intel Oct 24 1994; Intel initially downplayed it; public Usenet/CompuServe disclosure Oct 30 forced Intel's hand; $475M pre-tax charge announced Jan 17 1995 — the first full recall of a computer chip.

Saved hooks not followed:

• Vancouver Stock Exchange 1982 index-truncation error (index lost ~half its value over 22 months from systematic truncation-not-rounding on recalculation) — from general search context — reason saved: another vivid Linnainmaa-adjacent "accumulated error" case, but the chain had already found its strongest instance in Pentium/Nicely; would dilute rather than extend.
• Tim Coe's reverse-engineering of the Pentium's defective PD-plot — from Pratt 1995 — reason saved: a genuine mechanism deep-dive but would be a fifth zoom-in in a chain already well-shaped at 4 hops.
• Ariane 5 Flight 501 (1996) integer-overflow disaster — from general search context — reason saved: technically an overflow, not a rounding error; adjacent but category-mismatched to the seed's specific claim.

post-worthy: maybe — a clean, well-sourced "abstract math theory has literal $475M teeth" story, but it's one node in a broader "numerical-disaster" cluster (Patriot, Vancouver, Ariane) that would be more post-worthy assembled together than alone.