Because decode underutilizes GPU compute and power while prefill does not, LLM-inference services over-provision GPU hardware and cloud providers build additional datacenter capacity to meet the resulting demand

gpu-utilizationdatacenterpower-wallsplitwiseinference-servingcompute-economicsmemory-bandwidth

Patel et al. ("Splitwise," Microsoft/University of Washington, ISCA 2024) name a concrete industrial consequence of decode's memory-boundedness: because a single GPU generation must serve both the compute-bound prompt/prefill phase and the memory-bound token-generation/decode phase, production services "need to over-provision expensive GPUs to meet tight inference service level objectives (SLOs)" for interactive applications, and, at the industry level, "cloud service providers (CSPs) are having to build a lot of new datacenters to meet the GPU demand, and are running into a power wall." The paper's own framing states this as two of its named findings: "Insight VI: While the prompt phase utilizes the power budget of the GPU efficiently, the token phase does not," and "Insight VII: Token generation can be run on less compute-capable hardware for better Perf/W and Perf/$ efficiencies."

The paper's hardware comparison table supplies the concrete divergence driving this: from NVIDIA A100 to H100, the newer GPU has "3.43x more compute and 1.75x more power" than the A100, while HBM "bandwidth only grew by 1.6x, with no increase in memory" capacity — the same compute-vs-bandwidth scaling gap at hardware-generation scale. A single GPU tier is therefore well-matched to neither phase; decode "underutilizes compute resources" even under state-of-the-art batching. Splitwise's own proposed fix — disaggregating prefill and decode onto separately provisioned machines — is offered by its authors specifically to recover the efficiency this mismatch currently costs.