Peak AI hardware compute has scaled roughly 600x faster than memory bandwidth (and ~2,000x faster than interconnect bandwidth) over 20 years, making memory bandwidth the dominant AI-serving bottleneck

memory-wallmemory-bandwidthai-hardwarerooflinearithmetic-intensitygpuscaling-gap

Gholami et al. ("AI and Memory Wall," UC Berkeley/ICSI/LBNL, 2024) document a widening gap between two hardware growth curves over the past two decades: peak hardware FLOPS has scaled at roughly 3.0x every two years, while DRAM bandwidth scaled at only ~1.6x and interconnect bandwidth at only ~1.4x over the same period. In the paper's own aggregate figures, peak FLOPS increased 60,000x over the past 20 years against 100x for DRAM bandwidth and 30x for interconnect bandwidth — a divergence of roughly 600x versus DRAM and roughly 2,000x versus interconnect. The paper states plainly that "the main performance bottleneck is increasingly shifting to memory bandwidth" for both training and serving large models.

The paper's own case study grounds this in a concrete architectural split: encoder-style (BERT-like) models perform dense matrix-matrix operations and stay compute-bound, while decoder-style (GPT-like) autoregressive models perform repeated matrix-vector operations with orders-of-magnitude lower arithmetic intensity, "which makes it very challenging to effectively utilize a given hardware's compute units." Although the paper never uses the words "prefill" or "decode," this is the identical split the vault's own prefill/decode note describes under different vocabulary — decode's memory-boundedness is one named instance of an industry-wide scaling trend, not an isolated LLM-serving quirk. The roofline model is the general formal apparatus this instantiates, and Google's own TPU is a second, independent hardware case showing the same wall. Gholami et al. trace the observation further back than Wulf & McKee's 1994 naming, crediting John Ousterhout's 1990 operating-systems performance analysis as one of the earliest recorded observations of the memory-bandwidth problem.