Despite a purpose-built systolic matrix engine, four of Google's six production TPU neural-net workloads are memory-bandwidth-bound, not compute-bound

Google's 2017 TPU paper measured six of its own production neural-network applications (a mix of MLPs, CNNs, and LSTMs running in Google's data centers) against the chip's roofline performance model and reported that "four of the six NN applications are memory-bound." This is a specific, surprising quantitative finding about a chip whose defining feature is a 65,536-MAC systolic matrix multiply unit (claim-tpu-matrix-unit-called-heart-of-the-tpu) purpose-built for arithmetic throughput — yet most of the workloads run on it don't saturate that arithmetic capacity; they are limited instead by how fast data (weights, activations) can be moved to the compute units, per the roofline model's framing of operational intensity.

The finding generalizes a pattern this vault has already documented elsewhere in AI compute: transformer-inference decode is memory-bandwidth-bound for the same structural reason — one token's worth of arithmetic per pass through a large set of weights that must be re-read from memory each step (claim-llm-inference-prefill-decode). The TPU result shows the same wall appearing on hardware engineered from the ground up to avoid it, which is evidence that the memory-bandwidth constraint in modern AI workloads is closer to a structural property of the arithmetic-intensity ratio than an artifact of any one chip's design (see also claim-ertl-memory-wall-fallacy-application-dependent on why "memory-bound" is a workload property, not a universal law).