Capture: Does prefill/decode bridge to inference-dominant-compute? Memory bandwidth is the mechanism, not the magnitude

gap-detectionbridgeinferencememory-wallprefill-decodecompute-economicsgpu-utilizationarithmetic-intensity

Investigating the bipartite pair named in the task: claim-inference-dominant-ai-compute-2026 ("by 2026, inference is ~two-thirds of AI compute spend") and claim-llm-inference-prefill-decode ("decode is memory-bandwidth-bound, prefill is compute-bound"), cosine 0.86, no shared vocabulary.

Bridge-seed check performed first: 10-inbox/raw/2026-07-06-is-training-vs-inference-compute-asymmetry-the-missing-bridge-note-between-the-vaults-two-biggest-clusters.md already links both slugs (it names claim-llm-inference-prefill-decode in its related_notes), but its actual investigated bridge is a different one — training vs. inference FLOPs asymmetry via backpropagation's forward/backward pass (Epoch AI sourcing). It never opens the prefill/decode note's own mechanism content. This capture is not a duplicate of that one; it investigates the pairing the task actually names — whether the internal prefill/decode phase split explains the external claim about inference's compute-spend dominance — and treats the 2026-07-06 capture's findings (below) as standing context rather than re-deriving them.

Short answer: the resemblance is a real mechanism, not a false friend, but it is an indirect, two-hop one, and it does not rescue the vault's disputed "two-thirds" magnitude figure. Decode's low arithmetic intensity (the mechanism in the prefill/decode note) is a specific instance of a hardware-wide trend — compute (FLOPs) has scaled roughly 60,000× over 20 years while memory/interconnect bandwidth scaled only ~100–30× — and this same trend is what forces production LLM-serving deployments to over-provision GPU hardware and build additional datacenter capacity, independent of how few FLOPs any single inference actually costs. That is a genuine structural link between "how inference computes" and "how much hardware inference-as-an-industry consumes." It is not, however, a verification of the specific "two-thirds of all AI compute" figure, which two independent prior research passes in this vault (2026-07-06, twice) already traced to a chip-vendor CEO's advocacy op-ed and to a null result against Epoch AI / Stanford HAI primaries respectively.

Claim: The gap between how fast AI hardware compute and memory bandwidth have each scaled over the past 20 years is the structural reason memory bandwidth — not compute — has become the dominant AI serving bottleneck; decode's memory-boundedness is one named instance of this general trend

Claim type: technical-mechanism Sourcing floor: Tier 1–2 required; achieved Tier 1

"for serving/training LLMs. However, the main performance" … "bottleneck is increasingly shifting to memory bandwidth"

"peak hardware FLOPS has been scaling at" [3.0×/2yrs; DRAM bandwidth 1.6×/2yrs; interconnect bandwidth 1.4×/2yrs — figures given in the paper's abstract and Fig. 1]

"main limiting factoring in serving large AI models" [sic — "factoring" appears to be the paper's own wording, preserved as extracted]

Amir Gholami et al., "AI and Memory Wall," UC Berkeley / ICSI / LBNL, arXiv 2403.14123 (extended version of a paper published in IEEE Micro).

The paper's own case study names the mechanism the vault's prefill/decode note already carries at Tier 3: decoder-style (autoregressive) models perform repeated matrix-vector operations with orders of magnitude lower arithmetic intensity than matrix-matrix operations, "which makes it very challenging to effectively utilize a given hardware's compute units" — the same reason decode is memory-bandwidth-bound. This paper does not use the words "prefill" or "decode," but its own case study explicitly separates BERT-style (encoder, matrix-matrix) inference from GPT-style (decoder, matrix-vector, autoregressive) inference and shows the latter is memory-bound at low batch size — the identical split the vault's note describes under different vocabulary. The vault already has this general mechanism formalized via the roofline model and instantiated in hardware via Google's TPU; this source adds the industry-wide hardware-scaling-rate data (compute vs. bandwidth growth curves) that explains why the gap keeps widening rather than closing.

Provenance: https://arxiv.org/pdf/2403.14123 — Tier 1 (named authors, UC Berkeley/ICSI/LBNL, arXiv preprint of an IEEE Micro publication). source_sha: 6d86cdf96909d92500c23bdf90eed21641f12169d8d37eb9083bd97f11c6faa2.

Claim: Because the decode phase underutilizes GPU compute and power while the prefill phase does not, production LLM-inference services must over-provision GPU hardware to meet latency targets, and cloud providers are building additional datacenter capacity specifically to meet this GPU demand

Claim type: technical-mechanism (with supporting quantitative figures) Sourcing floor: Tier 1–2 required; achieved Tier 1

"need to over-provision expensive GPUs to meet tight inference" … "cloud service providers (CSPs) are having to" … "build a lot of new datacenters to meet the GPU demand, and" … "are running into a power wall"

"Insight VI: While the prompt phase utilizes the power budget" … "of the GPU efficiently, the token phase does not."

"Insight VII: Token generation can be run on less compute-" … "capable hardware for better Perf/W and Perf/$ efficiencies."

Pratyush Patel et al. (Microsoft / University of Washington), "Splitwise: Efficient Generative LLM Inference Using Phase Splitting," arXiv 2311.18677 (published ISCA 2024).

The paper's own hardware comparison table gives the concrete divergence driving this: from NVIDIA A100 to H100, the paper reports the GPUs "have 3.43× more compute and 1.75× more power compared" to A100, while HBM "bandwidth only grew by 1.6×, with no increase in memory" capacity. Because prompt/prefill processing is compute-bound and token-generation/decode is memory-bound, a single GPU generation serving both phases is well-matched to neither — decode "underutilizes compute resources" even under state-of-the-art batching, per the paper's own characterization. The paper's proposed fix (splitting prefill and decode onto separate, differently-provisioned machines) is offered specifically to recover the hardware efficiency lost to this mismatch, "to achieve better utilization, reduce hardware ownership costs, and save energy" — i.e., the phase split is presented by its own authors as a response to a real, current over-provisioning cost.

Provenance: https://arxiv.org/pdf/2311.18677 — Tier 1 (named authors, Microsoft/UW, arXiv preprint of an ISCA 2024 paper). source_sha: c7baf19039f7b760c5d83fb61324ec4ffdf384e8a047bd46f2c4e2ad85e3531d.

Claim: An independent, non-vendor Tier-1 systems paper states as a premise (not a measurement) that LLM inference's computational demand exceeds that of training, because of the volume of inference needed to amortize a model's training cost

Claim type: technical-mechanism-adjacent economic premise — recorded with an explicit epistemic caveat, see below Sourcing floor: Tier 1–2 required for a mechanism claim; source is Tier 1, but the specific sentence is unattributed within its own paper (no citation or measurement backs it) — recorded as a premise stated by domain experts in a peer-reviewed venue, not as a verified statistic

"computational demand for LLM inference far exceeds that" … "of training due to the vast number of applications leveraging" LLMs.

"a large number of" … "inferences are necessary to amortize the high training costs."

Same source as the previous claim (Patel et al., Splitwise, arXiv 2311.18677).

Why this matters and why it is not enough on its own: claim-inference-dominant-ai-compute-2026 currently carries a [unverified-quant — needs primary] flag, and two independent prior research passes in this vault (2026-07-06) traced its "two-thirds" figure to a chip-vendor CEO's World Economic Forum op-ed and separately found no primary industry-wide measurement at Epoch AI or Stanford HAI. This Splitwise sentence is the first non-vendor, peer-reviewed, Tier-1 source found in this session that independently asserts inference demand exceeds training demand — but it asserts direction, not magnitude, and it is not cited to any external data within the paper itself (it reads as accepted background premise for systems researchers, not a finding the paper set out to measure). It corroborates the qualitative shape of claim-inference-dominant-ai-compute-2026's framing without validating "two-thirds" or any other specific percentage. Recording this distinction explicitly rather than letting the Tier-1 source lend borrowed confidence to the un-sourced number.

Provenance: https://arxiv.org/pdf/2311.18677 — Tier 1 for authorship/venue; the specific claim is an uncited premise within that source. source_sha: c7baf19039f7b760c5d83fb61324ec4ffdf384e8a047bd46f2c4e2ad85e3531d.

Answering the core question

The two seed claims share no vocabulary because they describe the same physical constraint at two different scales. claim-llm-inference-prefill-decode describes it at the level of a single forward pass: decode's matrix-vector arithmetic has low arithmetic intensity, so the GPU spends most of its time waiting on memory, not computing. claim-inference-dominant-ai-compute-2026 describes it at the level of an industry: inference is claimed to consume a large and growing share of AI compute spend (dollars, hardware, datacenters) even though — per claim-training-inference-compute-asymmetry-mechanism, already established in this vault at Tier 1 — a single inference costs roughly twelve orders of magnitude fewer FLOPs than a full training run. Those two facts look contradictory until the mechanism found in this session connects them: if decode is memory-bound rather than compute-bound, then the FLOPs-based intuition ("inference is cheap, so it shouldn't dominate spend") is measuring the wrong resource. Gholami et al. establish the general trend (compute has scaled ~600× faster than memory bandwidth over 20 years) and Patel et al. establish the operational consequence at deployment scale (services "over-provision expensive GPUs" and CSPs "build a lot of new datacenters to meet the GPU demand" specifically because decode's memory-boundedness cannot be solved by adding more FLOPs). The bridge is real: inference's hardware/dollar footprint can grow disproportionately to its FLOPs share precisely because of the mechanism the prefill/decode note describes. What this session did not find, and does not claim to have found, is a primary measurement of how large that disproportion is industry-wide — the "two-thirds" figure remains exactly as unverified as the existing flags on claim-inference-dominant-ai-compute-2026 and myth-inference-two-thirds-of-compute already state. The mechanism-level bridge and the magnitude-level headline are, again, arriving from different evidentiary places — the same seam the 2026-07-06 capture found on the training-vs-inference side of this cluster.

Further leads

• "Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode" (arXiv 2605.30571) — title suggests a complication or refinement of the "decode is memory-bandwidth-bound" claim; not read this session, worth a follow-up hop before treating decode's memory-boundedness as settled beyond dispute.
• DistServe (Zhong et al., OSDI 2024) and SARATHI (arXiv 2308.16369) — other primary systems papers on prefill/decode disaggregation, not fetched this session; likely reinforce Claim 2 with independent corroboration and would help discharge the single-source concentration question if this branch grows.
• A secondary source (Towards Data Science, "Prefill Is Compute-Bound. Decode Is Memory-Bound.") reports concrete GPU-utilization figures — Llama-70B on H100 allegedly drops from 92% compute utilization in prefill to 30% in decode, and decode allegedly costs "as much as $200× prefill for batch size 1" — Tier 3, not fetched/verified directly, recorded as [unverified-quant — needs primary] lead only; would sharpen Claim 2 with an actual utilization number if traced to its origin.
• NVIDIA's own H100/A100 spec sheets (cited as reference [5] in the Splitwise paper's Table I) are the ultimate primary for the compute/bandwidth ratios quoted here at one remove; not fetched directly this session.

Safety flags

None. No page encountered during this session showed addressed-to-AI language, override language, claimed authority, tier self-assignment, file-system instructions, credential requests, or urgency framing. Both PDFs fetched via extract_pdf returned tls: "verified".

Entity candidates

• William Wulf — person — coined "memory wall" with Sally McKee in 1994/95; Gholami et al.'s entire framing explicitly measures itself against this prediction ("Later in 1995, William Wulf and Sally Mckee further echoed this prediction and coined the term 'memory wall'"). Already has a claim-note (claim-memory-wall-named-1994-wulf-mckee) but no entity page — the foundational figure this session's primary source is built on top of.
• Sally McKee — person — co-coined "memory wall" with Wulf; same priority note as above.
• John Ousterhout — person — Gholami et al. cite his 1990 OS-performance analysis as "one of the earliest observations" of the memory-bandwidth problem, older than Wulf & McKee's own 1994/95 naming. No vault note or entity page yet found.
• Samuel Williams — person — originator of the "arithmetic intensity" metric (roofline model, 2009) that both this session's primary source and the vault's existing claim-roofline-model-compute-bound-vs-memory-bound build on directly; appears as an author in that existing claim-note but has no standalone entity page.
• Amir Gholami — person — lead author, "AI and Memory Wall" (UC Berkeley); this session's primary source for the general memory-bandwidth-bottleneck mechanism.
• Pratyush Patel — person — lead author, "Splitwise" (University of Washington / Microsoft); this session's primary source for the GPU over-provisioning / datacenter-buildout mechanism.
• Splitwise — concept/term — the specific prefill/decode hardware-disaggregation technique; distinct enough from the general claim-llm-inference-prefill-decode mechanism claim to potentially warrant its own concept note if the disaggregation literature (DistServe, SARATHI, etc.) grows into a cluster.
• arithmetic intensity — concept — the FLOPs-per-byte-moved ratio that determines whether a workload is compute-bound or memory-bound; already implicit in claim-roofline-model-compute-bound-vs-memory-bound but not yet its own linkable term.