LLM inference splits into two distinct compute phases: prefill (compute-bound) and decode (memory-bandwidth-bound)

When a large language model runs inference, it does so in two mechanically distinct phases with different hardware bottlenecks, different latency profiles, and different user-experience consequences.

Prefill

"Prefill processes your entire input prompt at once, including system instructions, retrieved context, and the user message. It builds the internal state, called the key-value (KV) cache, that the model needs for generation." (Jim Allen Wallace, Redis, 2026-04-28)

Prefill is compute-bound: the GPU is constrained by how fast it can perform matrix multiplications over the full input sequence in parallel. All input tokens are processed simultaneously (unlike the sequential nature of decode). The latency produced by prefill is captured by the metric Time to First Token (TTFT) — the elapsed time before a user sees any output begin to stream.

Decode

"Decode generates the response one token at a time. Using the cached state from prefill, it produces a token, feeds it back in, and repeats until the response is complete." (Jim Allen Wallace, Redis, 2026-04-28)

Decode is memory-bandwidth-bound: the GPU is constrained not by compute speed but by how fast it can move the KV cache data from VRAM into compute units on every token step. "Every decode step has to read all of [the KV cache], which is a big reason decode is memory-bandwidth-bound." The relevant latency metric is Inter-Token Latency (ITL) — the pause between consecutive streamed tokens.

From the user's perspective, decode is the response streaming in: "low ITL feels like smooth typing while high ITL feels like pauses between words."

Performance diagnostic

Why this distinction matters

The two phases respond to different hardware optimizations. Inference hardware selection, batching strategy, and system architecture all depend on whether the workload is prefill-heavy (long-context retrieval) or decode-heavy (long output generation). The KV cache — the data structure that bridges the two phases — is itself a major source of memory pressure at scale. See claim-kv-cache-grows-with-context.

The Pragmatic Engineer (Gergely Orosz) summarizes the key metrics as: tokens per second (TPS) for throughput, TTFT for perceived responsiveness, and ITL for streaming smoothness — the three levers inference engineers tune in production. See claim-inference-engineering-emerged-as-specialty.

See also: claim-ai-inference-means-running-a-model, claim-kv-cache-grows-with-context