Dense associative memory uses a higher-order energy function to store more patterns than it has neurons, via a duality with deep-learning networks

Krotov & Hopfield (2016), "Dense Associative Memory for Pattern Recognition" (arXiv 1606.01164 — Tier 1), break the storage ceiling of the classical Hopfield network by replacing its quadratic energy with a higher-order (rectified-polynomial) energy function. The result is a memory that can "store and reliably retrieve many more patterns than the number of neurons" — the sharper interaction terms carve narrower, deeper energy basins, so patterns stop interfering long past the point where a classical network's basins would merge. Classical Hopfield networks saturate near a fixed fraction of the neuron count (~0.14N in the standard telling — held here as [unverified-quant], see question-verify-hopfield-classical-capacity-0138n-primary); the dense construction scales the capacity with the degree of the energy polynomial, reaching exponential storage in the limit.

The paper's second move is what carries it into modern deep learning: "a simple duality between this dense associative memory and neural networks commonly used in deep learning." Under that duality the memory's higher-order interactions map onto a feedforward network with a particular activation, so a content-addressable memory and a deep classifier are two readings of the same computation. This capacity result is the enabling precondition for the later claim that the continuous-state Hopfield update rule equals transformer attention: the equivalence is only interesting because dense memories store exponentially many patterns, so an attention layer over a large key set is a retrieval over a memory that actually has room for them.

The result is notable authorship as much as mechanism: John Hopfield revived and extended his own 1982 model here, rather than the breakthrough coming from outside (claim-hopfield-1982-energy-function-from-spin-glass-physics).