Dense associative memory uses a higher-order energy function to store more patterns than it has neurons, via a duality with deep-learning networks

hopfield-networksassociative-memorycapacitydeep-learningdmitry-krotov

Krotov & Hopfield (2016), "Dense Associative Memory for Pattern Recognition" (arXiv 1606.01164 — Tier 1), break the storage ceiling of the classical Hopfield network by replacing its quadratic energy with a higher-order (rectified-polynomial) energy function. The result is a memory that can "store and reliably retrieve many more patterns than the number of neurons" — the sharper interaction terms carve narrower, deeper energy basins, so patterns stop interfering long past the point where a classical network's basins would merge. Classical Hopfield networks saturate near a fixed fraction of the neuron count (~0.14N in the standard telling — held here as [unverified-quant], see question-verify-hopfield-classical-capacity-0138n-primary); the dense construction scales the capacity with the degree of the energy polynomial, reaching exponential storage in the limit [promotion wording; corrected 2026-09-12 against the paper — Krotov & Hopfield 2016 derive K^max = α_n·N^(n−1) (Eq. 5; error-free form Eq. 6): polynomial in N with the exponent set by the energy degree n, which they describe as capacity that "rapidly grows with N in a non-linear way"; the word "exponential" does not appear in the paper. Exponential-in-N capacity is the later result of Demircigil, Heusel, Löwe, Upgang & Vermet (2017, J. Stat. Phys. 168:288; arXiv:1702.01929), who take the degree to infinity — an exponential interaction function — and "prove that model has an exponential storage capacity in the number of neurons"; the 2020 attention-equivalence paper's "exponentially many patterns" rests on that line, not on the 2016 paper alone].

The paper's second move is what carries it into modern deep learning: "a simple duality between this dense associative memory and neural networks commonly used in deep learning." Under that duality the memory's higher-order interactions map onto a feedforward network with a particular activation, so a content-addressable memory and a deep classifier are two readings of the same computation. This capacity result is the enabling precondition for the later claim that the continuous-state Hopfield update rule equals transformer attention: the equivalence is only interesting because dense memories store exponentially many patterns, so an attention layer over a large key set is a retrieval over a memory that actually has room for them.

The result is notable authorship as much as mechanism: John Hopfield revived and extended his own 1982 model here, rather than the breakthrough coming from outside (claim-hopfield-1982-energy-function-from-spin-glass-physics).