---
title: "Peak AI hardware compute has scaled roughly 600x faster than memory bandwidth (and ~2,000x faster than interconnect bandwidth) over 20 years, making memory bandwidth the dominant AI-serving bottleneck"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
audit_status: "capture-verified; 2026-08-13 scheduled audit (claude-fable-5, cross-model): corrected — body's 'aggregate divergence on the order of 100–600x' misread the paper's figures. Per the PDF (re-extracted, sha match): peak FLOPS +60,000x over 20 years vs. 100x (DRAM BW) and 30x (interconnect BW), i.e. divergence ~600x vs. DRAM and ~2,000x vs. interconnect. Title's lumping of memory+interconnect under one 600x figure adjusted to match. All source quotes re-verified verbatim against the PDF."
source_url: "https://arxiv.org/pdf/2403.14123"
source_title: "AI and Memory Wall"
source_author: "Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer (UC Berkeley / ICSI / LBNL)"
source_date: "2024-03-21"
source_tier: 1
source_sha: "6d86cdf96909d92500c23bdf90eed21641f12169d8d37eb9083bd97f11c6faa2"
source_quote: "the main performance bottleneck is increasingly shifting to memory bandwidth"
provenance: "Promotion from 10-inbox/raw/2026-08-12-bipartite-two-things-this-vault-knows-in-different.md, 2026-08-12"
origin: "batch"
derived_from: "10-inbox/raw/2026-08-12-bipartite-two-things-this-vault-knows-in-different.md"
date_created: "2026-08-12T00:00:00.000Z"
tags: ["memory-wall","memory-bandwidth","ai-hardware","roofline","arithmetic-intensity","gpu","scaling-gap"]
related_notes: ["claim-roofline-model-compute-bound-vs-memory-bound","claim-tpu-most-nn-workloads-memory-bandwidth-bound","claim-memory-wall-named-1994-wulf-mckee","claim-llm-inference-prefill-decode"]
seek_code_commit: "729ee25"
---


Gholami et al. ("AI and Memory Wall," UC Berkeley/ICSI/LBNL, 2024) document a widening gap between two hardware growth curves over the past two decades: peak hardware FLOPS has scaled at roughly 3.0x every two years, while DRAM bandwidth scaled at only ~1.6x and interconnect bandwidth at only ~1.4x over the same period. In the paper's own aggregate figures, peak FLOPS increased 60,000x over the past 20 years against 100x for DRAM bandwidth and 30x for interconnect bandwidth — a divergence of roughly 600x versus DRAM and roughly 2,000x versus interconnect. The paper states plainly that "the main performance bottleneck is increasingly shifting to memory bandwidth" for both training and serving large models.

The paper's own case study grounds this in a concrete architectural split: encoder-style (BERT-like) models perform dense matrix-matrix operations and stay compute-bound, while decoder-style (GPT-like) autoregressive models perform repeated matrix-vector operations with orders-of-magnitude lower arithmetic intensity, "which makes it very challenging to effectively utilize a given hardware's compute units." Although the paper never uses the words "prefill" or "decode," this is the identical split [[claim-llm-inference-prefill-decode|the vault's own prefill/decode note]] describes under different vocabulary — decode's memory-boundedness is one named instance of an industry-wide scaling trend, not an isolated LLM-serving quirk. The [[claim-roofline-model-compute-bound-vs-memory-bound|roofline model]] is the general formal apparatus this instantiates, and [[claim-tpu-most-nn-workloads-memory-bandwidth-bound|Google's own TPU]] is a second, independent hardware case showing the same wall. Gholami et al. trace the observation further back than [[claim-memory-wall-named-1994-wulf-mckee|Wulf & McKee's 1994 naming]], crediting John Ousterhout's 1990 operating-systems performance analysis as one of the earliest recorded observations of the memory-bandwidth problem.

> [!note] Seek's commentary:
> What earns this one a place isn't the number — 600x is a big, satisfying number, and big numbers are exactly the ones to distrust reflexively — it's that the paper's own case study rediscovers the vault's prefill/decode split without ever reaching for that vocabulary. Two literatures naming the same wall from opposite directions is a better corroboration than one literature restating its own finding loudly.
> — Seek