---
title: "Because decode underutilizes GPU compute and power while prefill does not, LLM-inference services over-provision GPU hardware and cloud providers build additional datacenter capacity to meet the resulting demand"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
audit_status: "capture-verified; 2026-08-13 scheduled audit (claude-fable-5, cross-model): corrected — source_quote was a non-verbatim splice: it read 'to meet tight inference latency targets...' where the paper (§I, re-extracted, sha match) reads 'to meet tight inference service level objectives (SLOs) for interactive applications. At the same time, ...'. Quote replaced with the verbatim passage and body wording aligned. Insights VI/VII and the A100→H100 figures (3.43x compute, 1.75x power, 1.6x HBM bandwidth, no capacity increase) re-verified verbatim."
source_url: "https://arxiv.org/pdf/2311.18677"
source_title: "Splitwise: Efficient Generative LLM Inference Using Phase Splitting"
source_author: "Pratyush Patel (University of Washington), Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini (Microsoft)"
source_date: "2024-05-20"
source_tier: 1
source_sha: "c7baf19039f7b760c5d83fb61324ec4ffdf384e8a047bd46f2c4e2ad85e3531d"
source_quote: "need to over-provision expensive GPUs to meet tight inference service level objectives (SLOs) for interactive applications. At the same time, cloud service providers (CSPs) are having to build a lot of new datacenters to meet the GPU demand, and are running into a power wall"
provenance: "Promotion from 10-inbox/raw/2026-08-12-bipartite-two-things-this-vault-knows-in-different.md, 2026-08-12"
origin: "batch"
derived_from: "10-inbox/raw/2026-08-12-bipartite-two-things-this-vault-knows-in-different.md"
date_created: "2026-08-12T00:00:00.000Z"
tags: ["gpu-utilization","datacenter","power-wall","splitwise","inference-serving","compute-economics","memory-bandwidth"]
related_notes: ["claim-llm-inference-prefill-decode","claim-compute-outpaces-memory-bandwidth-ai-hardware-scaling","claim-inference-dominant-ai-compute-2026","claim-kv-cache-grows-with-context"]
seek_code_commit: "729ee25"
---


Patel et al. ("Splitwise," Microsoft/University of Washington, ISCA 2024) name a concrete industrial consequence of [[claim-llm-inference-prefill-decode|decode's memory-boundedness]]: because a single GPU generation must serve both the compute-bound prompt/prefill phase and the memory-bound token-generation/decode phase, production services "need to over-provision expensive GPUs to meet tight inference service level objectives (SLOs)" for interactive applications, and, at the industry level, "cloud service providers (CSPs) are having to build a lot of new datacenters to meet the GPU demand, and are running into a power wall." The paper's own framing states this as two of its named findings: "Insight VI: While the prompt phase utilizes the power budget of the GPU efficiently, the token phase does not," and "Insight VII: Token generation can be run on less compute-capable hardware for better Perf/W and Perf/$ efficiencies."

The paper's hardware comparison table supplies the concrete divergence driving this: from NVIDIA A100 to H100, the newer GPU has "3.43x more compute and 1.75x more power" than the A100, while HBM "bandwidth only grew by 1.6x, with no increase in memory" capacity — the same [[claim-compute-outpaces-memory-bandwidth-ai-hardware-scaling|compute-vs-bandwidth scaling gap]] at hardware-generation scale. A single GPU tier is therefore well-matched to neither phase; decode "underutilizes compute resources" even under state-of-the-art batching. Splitwise's own proposed fix — disaggregating prefill and decode onto separately provisioned machines — is offered by its authors specifically to recover the efficiency this mismatch currently costs.

> [!note] Seek's commentary:
> This is the note that turns an architecture footnote into an economics claim: a phase split inside a single forward pass, multiplied across a fleet, becomes a reason cloud providers are pouring concrete. The FLOPs-based intuition that inference should be cheap because a token costs almost nothing measures the wrong resource once you're renting racks instead of running one request.
> — Seek