---
id: "20260812-0201-bipartite-two-things-this"
title: "Capture: Does prefill/decode bridge to inference-dominant-compute? Memory bandwidth is the mechanism, not the magnitude"
type: "capture"
status: "promoted"
origin: "batch"
promoted_to: ["30-notes/claim-compute-outpaces-memory-bandwidth-ai-hardware-scaling.md","30-notes/claim-decode-underutilization-forces-gpu-overprovisioning-datacenter-buildout.md","30-notes/claim-splitwise-uncited-premise-inference-demand-exceeds-training.md","30-notes/observation-prefill-decode-bridges-inference-dominant-compute-mechanism-not-magnitude.md","40-entities/entity-william-wulf.md","40-entities/entity-sally-mckee.md","40-entities/entity-samuel-williams.md"]
not_promoted: ["Further-leads item: arXiv 2605.30571 ('Memory-Bound but Not Bandwidth-Limited...') — title-only, not read this session; left as an unread lead, not a claim.","Further-leads item: DistServe (OSDI 2024) and SARATHI (arXiv 2308.16369) — not fetched this session; likely corroboration for the over-provisioning claim if this branch grows, but recording them now would be inventing a source.","Further-leads item: Towards Data Science secondary GPU-utilization figures (92%->30%, '$200x prefill') — Tier 3, not fetched/verified; capture itself flagged [unverified-quant -- needs primary], correctly left unpromoted.","Further-leads item: NVIDIA H100/A100 spec sheets — cited only at one remove via Splitwise's Table I; not fetched directly this session.","Entity candidate: John Ousterhout — real and on-topic (Gholami et al. credit his 1990 OS-performance work as an earlier observation of the memory-bandwidth problem than Wulf & McKee's 1994 naming), but this is a second-order citation (Gholami cites him; his own 1990 paper was not read this session) with no claim-note of his own to hang a hub's References on. Mentioned by name in claim-compute-outpaces-memory-bandwidth-ai-hardware-scaling.md's body instead of given a page — a hub with an empty References list is the dead-stub flood the entity spec warns against.","Entity candidates: Amir Gholami, Pratyush Patel — real, named lead authors of this session's two primaries, but single-paper citations with no independent recurring salience in the vault beyond authorship; skipped per the entity spec's bias against one-hub-per-paper flood. Named in full in each claim-note's source_author field instead.","Entity candidate: Splitwise (concept/term) — the capture's own text hedges this ('potentially warrant its own concept note if the disaggregation literature grows into a cluster'); a single-paper technique name, not yet recurring. Skipped; the technique is described inline in claim-decode-underutilization-forces-gpu-overprovisioning-datacenter-buildout.md instead.","Entity candidate: arithmetic intensity (concept) — already effectively hubbed by the existing claim-roofline-model-compute-bound-vs-memory-bound.md, whose own commentary calls itself 'worth treating as a hub note for the cluster.' A separate 40-entities/ page for the same concept would duplicate rather than consolidate the vault's link graph; skipped."]
writer_model: "claude-sonnet-5"
date_created: "2026-08-12T00:00:00.000Z"
provenance: "batch run 2026-08-12, bipartite hop-protocol task linking claim-inference-dominant-ai-compute-2026 and claim-llm-inference-prefill-decode (cosine 0.86, no shared vocabulary). Researched via WebSearch + mcp__seek__extract_pdf against arXiv-hosted primary systems-research papers. All quotes below were fetched directly via extract_pdf in this session (sha256 recorded per source), not relayed from memory or a summarizing layer."
derived_from: []
tags: ["gap-detection","bridge","inference","memory-wall","prefill-decode","compute-economics","gpu-utilization","arithmetic-intensity"]
source_url: "https://arxiv.org/pdf/2403.14123"
source_sha: "6d86cdf96909d92500c23bdf90eed21641f12169d8d37eb9083bd97f11c6faa2"
source_title: "AI and Memory Wall"
source_author: "Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, Kurt Keutzer (UC Berkeley / ICSI / LBNL)"
source_date: "2024-03-21 (arXiv v1; extended version of a paper published in IEEE Micro)"
source_venue: "arXiv (cs.LG), extended from IEEE Micro Journal"
source_tier: 1
other_sources: [{"url":"https://arxiv.org/pdf/2311.18677","sha":"c7baf19039f7b760c5d83fb61324ec4ffdf384e8a047bd46f2c4e2ad85e3531d","title":"Splitwise: Efficient Generative LLM Inference Using Phase Splitting","author":"Pratyush Patel (University of Washington), Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini (Microsoft)","date":"2024-05-20 (arXiv v2; published ISCA 2024)","note":"Load-bearing for the GPU over-provisioning / datacenter-buildout claim and the A100-vs-H100 compute/bandwidth divergence figures. Own venue (arXiv preprint of a peer-reviewed ISCA paper), fetched directly, tls verified.","tier":1}]
related_notes: ["claim-llm-inference-prefill-decode","claim-inference-dominant-ai-compute-2026","claim-roofline-model-compute-bound-vs-memory-bound","claim-tpu-most-nn-workloads-memory-bandwidth-bound","claim-memory-wall-named-1994-wulf-mckee","claim-training-inference-compute-asymmetry-mechanism","myth-inference-two-thirds-of-compute","claim-kv-cache-grows-with-context","moc-inference-economics"]
seek_code_commit: "729ee25"
---


Investigating the bipartite pair named in the task: [[claim-inference-dominant-ai-compute-2026]] ("by 2026, inference is ~two-thirds of AI compute spend") and [[claim-llm-inference-prefill-decode]] ("decode is memory-bandwidth-bound, prefill is compute-bound"), cosine 0.86, no shared vocabulary.

**Bridge-seed check performed first:** `10-inbox/raw/2026-07-06-is-training-vs-inference-compute-asymmetry-the-missing-bridge-note-between-the-vaults-two-biggest-clusters.md` already links both slugs (it names `claim-llm-inference-prefill-decode` in its `related_notes`), but its actual investigated bridge is a *different* one — [[training]] vs. [[inference]] FLOPs asymmetry via backpropagation's forward/backward pass (Epoch AI sourcing). It never opens the prefill/decode note's own mechanism content. This capture is not a duplicate of that one; it investigates the pairing the task actually names — whether the *internal* prefill/decode phase split explains the *external* claim about inference's compute-spend dominance — and treats the 2026-07-06 capture's findings (below) as standing context rather than re-deriving them.

**Short answer:** the resemblance is a real mechanism, not a false friend, but it is an indirect, two-hop one, and it does not rescue the vault's disputed "two-thirds" magnitude figure. Decode's low arithmetic intensity (the mechanism in the prefill/decode note) is a specific instance of a hardware-wide trend — compute (FLOPs) has scaled roughly 60,000× over 20 years while memory/interconnect bandwidth scaled only ~100–30× — and this same trend is what forces production LLM-serving deployments to over-provision GPU hardware and build additional datacenter capacity, independent of how few FLOPs any single inference actually costs. That is a genuine structural link between "how inference computes" and "how much hardware inference-as-an-industry consumes." It is not, however, a verification of the specific "two-thirds of all AI compute" figure, which two independent prior research passes in this vault (2026-07-06, twice) already traced to a chip-vendor CEO's advocacy op-ed and to a null result against Epoch AI / Stanford HAI primaries respectively.

---

## Claim: The gap between how fast AI hardware compute and memory bandwidth have each scaled over the past 20 years is the structural reason memory bandwidth — not compute — has become the dominant AI serving bottleneck; decode's memory-boundedness is one named instance of this general trend

**Claim type:** technical-mechanism
**Sourcing floor:** Tier 1–2 required; achieved Tier 1

> "for serving/training LLMs. However, the main performance" … "bottleneck is increasingly shifting to memory bandwidth"

> "peak hardware FLOPS has been scaling at" [3.0×/2yrs; DRAM bandwidth 1.6×/2yrs; interconnect bandwidth 1.4×/2yrs — figures given in the paper's abstract and Fig. 1]

> "main limiting factoring in serving large AI models" [sic — "factoring" appears to be the paper's own wording, preserved as extracted]

Amir Gholami et al., "AI and Memory Wall," UC Berkeley / ICSI / LBNL, arXiv 2403.14123 (extended version of a paper published in IEEE Micro).

The paper's own case study names the mechanism the vault's prefill/decode note already carries at Tier 3: decoder-style (autoregressive) models perform repeated matrix-vector operations with orders of magnitude lower arithmetic intensity than matrix-matrix operations, "which makes it very challenging to effectively utilize a given hardware's compute units" — the same reason [[claim-llm-inference-prefill-decode|decode is memory-bandwidth-bound]]. This paper does not use the words "prefill" or "decode," but its own case study explicitly separates BERT-style (encoder, matrix-matrix) inference from GPT-style (decoder, matrix-vector, autoregressive) inference and shows the latter is memory-bound at low batch size — the identical split the vault's note describes under different vocabulary. The vault already has this general mechanism formalized via the [[claim-roofline-model-compute-bound-vs-memory-bound|roofline model]] and instantiated in hardware via [[claim-tpu-most-nn-workloads-memory-bandwidth-bound|Google's TPU]]; this source adds the industry-wide hardware-scaling-rate data (compute vs. bandwidth growth curves) that explains *why* the gap keeps widening rather than closing.

**Provenance:** https://arxiv.org/pdf/2403.14123 — Tier 1 (named authors, UC Berkeley/ICSI/LBNL, arXiv preprint of an IEEE Micro publication). source_sha: 6d86cdf96909d92500c23bdf90eed21641f12169d8d37eb9083bd97f11c6faa2.

---

## Claim: Because the decode phase underutilizes GPU compute and power while the prefill phase does not, production LLM-inference services must over-provision GPU hardware to meet latency targets, and cloud providers are building additional datacenter capacity specifically to meet this GPU demand

**Claim type:** technical-mechanism (with supporting quantitative figures)
**Sourcing floor:** Tier 1–2 required; achieved Tier 1

> "need to over-provision expensive GPUs to meet tight inference" … "cloud service providers (CSPs) are having to" … "build a lot of new datacenters to meet the GPU demand, and" … "are running into a power wall"

> "Insight VI: While the prompt phase utilizes the power budget" … "of the GPU efficiently, the token phase does not."

> "Insight VII: Token generation can be run on less compute-" … "capable hardware for better Perf/W and Perf/$ efficiencies."

Pratyush Patel et al. (Microsoft / University of Washington), "Splitwise: Efficient Generative LLM Inference Using Phase Splitting," arXiv 2311.18677 (published ISCA 2024).

The paper's own hardware comparison table gives the concrete divergence driving this: from NVIDIA A100 to H100, the paper reports the GPUs "have 3.43× more compute and 1.75× more power compared" to A100, while HBM "bandwidth only grew by 1.6×, with no increase in memory" capacity. Because prompt/prefill processing is compute-bound and token-generation/decode is memory-bound, a single GPU generation serving both phases is well-matched to neither — decode "underutilizes compute resources" even under state-of-the-art batching, per the paper's own characterization. The paper's proposed fix (splitting prefill and decode onto separate, differently-provisioned machines) is offered specifically to recover the hardware efficiency lost to this mismatch, "to achieve better utilization, reduce hardware ownership costs, and save energy" — i.e., the phase split is presented by its own authors as a response to a real, current over-provisioning cost.

**Provenance:** https://arxiv.org/pdf/2311.18677 — Tier 1 (named authors, Microsoft/UW, arXiv preprint of an ISCA 2024 paper). source_sha: c7baf19039f7b760c5d83fb61324ec4ffdf384e8a047bd46f2c4e2ad85e3531d.

---

## Claim: An independent, non-vendor Tier-1 systems paper states as a premise (not a measurement) that LLM inference's computational demand exceeds that of training, because of the volume of inference needed to amortize a model's training cost

**Claim type:** technical-mechanism-adjacent economic premise — recorded with an explicit epistemic caveat, see below
**Sourcing floor:** Tier 1–2 required for a mechanism claim; source is Tier 1, but the specific sentence is unattributed within its own paper (no citation or measurement backs it) — recorded as a premise stated by domain experts in a peer-reviewed venue, not as a verified statistic

> "computational demand for LLM inference far exceeds that" … "of training due to the vast number of applications leveraging" LLMs.

> "a large number of" … "inferences are necessary to amortize the high training costs."

Same source as the previous claim (Patel et al., Splitwise, arXiv 2311.18677).

**Why this matters and why it is not enough on its own:** [[claim-inference-dominant-ai-compute-2026]] currently carries a `[unverified-quant — needs primary]` flag, and two independent prior research passes in this vault (2026-07-06) traced its "two-thirds" figure to a chip-vendor CEO's World Economic Forum op-ed and separately found no primary industry-wide measurement at Epoch AI or Stanford HAI. This Splitwise sentence is the first *non-vendor, peer-reviewed, Tier-1* source found in this session that independently asserts inference demand exceeds training demand — but it asserts direction, not magnitude, and it is not cited to any external data within the paper itself (it reads as accepted background premise for systems researchers, not a finding the paper set out to measure). It corroborates the qualitative shape of [[claim-inference-dominant-ai-compute-2026]]'s framing without validating "two-thirds" or any other specific percentage. Recording this distinction explicitly rather than letting the Tier-1 source lend borrowed confidence to the un-sourced number.

**Provenance:** https://arxiv.org/pdf/2311.18677 — Tier 1 for authorship/venue; the specific claim is an uncited premise within that source. source_sha: c7baf19039f7b760c5d83fb61324ec4ffdf384e8a047bd46f2c4e2ad85e3531d.

---

## Answering the core question

The two seed claims share no vocabulary because they describe the same physical constraint at two different scales. [[claim-llm-inference-prefill-decode]] describes it at the level of a single forward pass: decode's matrix-vector arithmetic has low arithmetic intensity, so the GPU spends most of its time waiting on memory, not computing. [[claim-inference-dominant-ai-compute-2026]] describes it at the level of an industry: inference is claimed to consume a large and growing share of AI compute *spend* (dollars, hardware, datacenters) even though — per [[claim-training-inference-compute-asymmetry-mechanism]], already established in this vault at Tier 1 — a single inference costs roughly twelve orders of magnitude fewer FLOPs than a full training run. Those two facts look contradictory until the mechanism found in this session connects them: if decode is memory-bound rather than compute-bound, then the FLOPs-based intuition ("inference is cheap, so it shouldn't dominate spend") is measuring the wrong resource. Gholami et al. establish the general trend (compute has scaled ~600× faster than memory bandwidth over 20 years) and Patel et al. establish the operational consequence at deployment scale (services "over-provision expensive GPUs" and CSPs "build a lot of new datacenters to meet the GPU demand" specifically because decode's memory-boundedness cannot be solved by adding more FLOPs). The bridge is real: inference's hardware/dollar footprint can grow disproportionately to its FLOPs share precisely because of the mechanism the prefill/decode note describes. What this session did not find, and does not claim to have found, is a primary measurement of *how large* that disproportion is industry-wide — the "two-thirds" figure remains exactly as unverified as the existing flags on [[claim-inference-dominant-ai-compute-2026]] and [[myth-inference-two-thirds-of-compute]] already state. The mechanism-level bridge and the magnitude-level headline are, again, arriving from different evidentiary places — the same seam the 2026-07-06 capture found on the training-vs-inference side of this cluster.

> [!note] Seek's commentary:
> The tidy version of this finding is "memory bandwidth is why inference is expensive to serve at scale, even though it's cheap in FLOPs" — but the honest version has to hold two things apart that are easy to blur: a *mechanism* that structurally favors inference consuming disproportionate hardware (well-sourced, Tier 1, confirmed here) and a *specific number* claiming exactly how disproportionate (still unsourced, after three separate research passes across two capture sessions have now failed to find it). The temptation with a bridge this clean is to let the mechanism's credibility rub off on the number. It shouldn't. If anything, a well-sourced mechanism for *why* the number could be large makes it more important, not less, to keep demanding a primary measurement of what the number actually is.

---

## Further leads

- "Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode" (arXiv 2605.30571) — title suggests a complication or refinement of the "decode is memory-bandwidth-bound" claim; not read this session, worth a follow-up hop before treating decode's memory-boundedness as settled beyond dispute.
- DistServe (Zhong et al., OSDI 2024) and SARATHI (arXiv 2308.16369) — other primary systems papers on prefill/decode disaggregation, not fetched this session; likely reinforce Claim 2 with independent corroboration and would help discharge the single-source concentration question if this branch grows.
- A secondary source (Towards Data Science, "Prefill Is Compute-Bound. Decode Is Memory-Bound.") reports concrete GPU-utilization figures — Llama-70B on H100 allegedly drops from 92% compute utilization in prefill to 30% in decode, and decode allegedly costs "as much as $200× prefill for batch size 1" — Tier 3, not fetched/verified directly, recorded as `[unverified-quant — needs primary]` lead only; would sharpen Claim 2 with an actual utilization number if traced to its origin.
- NVIDIA's own H100/A100 spec sheets (cited as reference [5] in the Splitwise paper's Table I) are the ultimate primary for the compute/bandwidth ratios quoted here at one remove; not fetched directly this session.

## Safety flags

None. No page encountered during this session showed addressed-to-AI language, override language, claimed authority, tier self-assignment, file-system instructions, credential requests, or urgency framing. Both PDFs fetched via `extract_pdf` returned `tls: "verified"`.

## Entity candidates

- William Wulf — person — coined "memory wall" with Sally McKee in 1994/95; Gholami et al.'s entire framing explicitly measures itself against this prediction ("Later in 1995, William Wulf and Sally Mckee further echoed this prediction and coined the term 'memory wall'"). Already has a claim-note ([[claim-memory-wall-named-1994-wulf-mckee]]) but no entity page — the foundational figure this session's primary source is built on top of.
- Sally McKee — person — co-coined "memory wall" with Wulf; same priority note as above.
- John Ousterhout — person — Gholami et al. cite his 1990 OS-performance analysis as "one of the earliest observations" of the memory-bandwidth problem, older than Wulf & McKee's own 1994/95 naming. No vault note or entity page yet found.
- Samuel Williams — person — originator of the "arithmetic intensity" metric (roofline model, 2009) that both this session's primary source and the vault's existing [[claim-roofline-model-compute-bound-vs-memory-bound]] build on directly; appears as an author in that existing claim-note but has no standalone entity page.
- Amir Gholami — person — lead author, "AI and Memory Wall" (UC Berkeley); this session's primary source for the general memory-bandwidth-bottleneck mechanism.
- Pratyush Patel — person — lead author, "Splitwise" (University of Washington / Microsoft); this session's primary source for the GPU over-provisioning / datacenter-buildout mechanism.
- Splitwise — concept/term — the specific prefill/decode hardware-disaggregation technique; distinct enough from the general [[claim-llm-inference-prefill-decode]] mechanism claim to potentially warrant its own concept note if the disaggregation literature (DistServe, SARATHI, etc.) grows into a cluster.
- arithmetic intensity — concept — the FLOPs-per-byte-moved ratio that determines whether a workload is compute-bound or memory-bound; already implicit in [[claim-roofline-model-compute-bound-vs-memory-bound]] but not yet its own linkable term.