---
title: "LLM inference splits into two distinct compute phases: prefill (compute-bound) and decode (memory-bandwidth-bound)"
type: "claim"
status: "seedling"
audit_status: "flagged (V-007)"
flags: ["[unverified-mechanism — needs primary] Prefill/decode bottleneck mechanism rests on a Tier-3 vendor blog (quotes verified verbatim against the source, but sources.md requires Tier 1-2 for technical-mechanism claims). Retro-fitted 2026-07-06 per Cali ruling 7 (audit V-007)."]
date_created: "2026-06-04T00:00:00.000Z"
provenance: "Seek research batch, 2026-06-04"
tags: ["LLM","inference","prefill","decode","KV-cache","GPU","latency","throughput"]
source_url: "https://redis.io/blog/prefill-vs-decode/"
source_title: "Prefill vs Decode: LLM Inference Phases Explained"
source_author: "Jim Allen Wallace"
source_date: "2026-04-28"
source_tier: 3
related_notes: ["claim-ai-inference-means-running-a-model","claim-kv-cache-grows-with-context","claim-inference-engineering-emerged-as-specialty"]
drafted_in: ["2026-07-13-the-heart-was-always-a-pump","from-a-standing-start","the-heart-was-always-a-pump"]
---


When a [[large language model]] runs inference, it does so in two mechanically distinct phases with different hardware bottlenecks, different latency profiles, and different user-experience consequences.

## Prefill

"Prefill processes your entire input prompt at once, including system instructions, retrieved context, and the user message. It builds the internal state, called the key-value (KV) cache, that the model needs for generation." (Jim Allen Wallace, Redis, 2026-04-28)

Prefill is **compute-bound**: the [[GPU]] is constrained by how fast it can perform [[matrix multiplication|matrix multiplications]] over the full input sequence in parallel. All input tokens are processed simultaneously (unlike the sequential nature of decode). The latency produced by prefill is captured by the metric **Time to First Token (TTFT)** — the elapsed time before a user sees any output begin to stream.

## Decode

"Decode generates the response one token at a time. Using the cached state from prefill, it produces a token, feeds it back in, and repeats until the response is complete." (Jim Allen Wallace, Redis, 2026-04-28)

Decode is **memory-bandwidth-bound**: the GPU is constrained not by compute speed but by how fast it can move the [[KV cache]] data from [[VRAM]] into compute units on every token step. "Every decode step has to read all of [the KV cache], which is a big reason decode is memory-bandwidth-bound." The relevant latency metric is **Inter-Token Latency (ITL)** — the pause between consecutive streamed tokens.

From the user's perspective, decode is *the response streaming in*: "low ITL feels like smooth typing while high ITL feels like pauses between words."

## Performance diagnostic

| Symptom | Likely bottleneck |
|---|---|
| Long wait before first word | Prefill (long prompt, compute-bound) |
| Slow, choppy streaming | Decode (memory-bandwidth-bound) |

## Why this distinction matters

The two phases respond to different hardware optimizations. Inference hardware selection, batching strategy, and system architecture all depend on whether the workload is prefill-heavy (long-context retrieval) or decode-heavy (long output generation). The KV cache — the data structure that bridges the two phases — is itself a major source of memory pressure at scale. See [[claim-kv-cache-grows-with-context]].

The Pragmatic Engineer (Gergely Orosz) summarizes the key metrics as: **tokens per second (TPS)** for throughput, **TTFT** for perceived responsiveness, and **ITL** for streaming smoothness — the three levers inference engineers tune in production. See [[claim-inference-engineering-emerged-as-specialty]].

See also: [[claim-ai-inference-means-running-a-model]], [[claim-kv-cache-grows-with-context]]