---
id: "20260706-1022-is-training-vs-inference"
title: "Capture: Is \"training-vs-inference compute asymmetry\" the missing bridge note between the vault's two biggest clusters?"
type: "capture"
status: "promoted"
date_promoted: "2026-07-07T00:00:00.000Z"
promoted_to: ["30-notes/claim-training-inference-compute-asymmetry-mechanism.md","30-notes/myth-inference-two-thirds-of-compute.md"]
not_promoted: ["Claim 4 (comparable-allocation theory) and Claim 5 (OpenAI 2024 split) — folded as context into both promoted notes rather than standing alone; Claim 5's chain-of-custody caveat preserved verbatim in the myth entry","R&D-vs-training-compute lead (Denain & Wu) — genuinely new Tier-1 source, left for the dominant-compute re-source run (topic already queued from remediation ruling 7)"]
origin: "batch"
model: "claude-sonnet-5"
date_created: "2026-07-06T00:00:00.000Z"
provenance: "batch run 2026-07-06, in answer to the open question 50-questions/question-bridge-training-inference-compute.md. Researched via WebSearch + WebFetch against Epoch AI's own blog/data-insights venue (primary compute-economics research org), plus a citation-chain trace on the vault's existing 'two-thirds of AI compute' figure back to its root source. All quotes below were fetched directly from the cited URL in this session, not relayed from memory."
derived_from: []
tags: ["gap-detection","missing-bridge","inference","backpropagation","compute-economics","epoch-ai","training-compute","forward-pass","backward-pass"]
tags_note: "mirrors the parent question's tags plus new sourcing-specific tags"
source_url: "https://epoch.ai/blog/trading-off-compute-in-training-and-inference"
source_author: "Pablo Villalobos and David Atkinson"
source_date: "2023-07-28"
source_tier: 1
other_sources: [{"url":"https://epoch.ai/blog/optimally-allocating-compute-between-inference-and-training","author":"Ege Erdil","date":"2024-03-29","note":"Load-bearing for the forward/backward-pass mechanism claim and the 3x token-cost multiplier. Epoch AI's own analytical derivation, fetched directly.","tier":1},{"url":"https://epoch.ai/data-insights/openai-compute-spend","author":"Josh You (Epoch AI)","date":"2025-10-10","note":"OpenAI 2024 $ compute-spend breakdown. Epoch AI's own dataset entry, but the underlying dollar figures are explicitly sourced by the article itself to The Information and The New York Times reporting, not to OpenAI's own disclosure — recorded with that caveat, not laundered to Tier 1.","tier":2},{"url":"https://epoch.ai/gradient-updates/r-and-d-vs-training-compute","author":"Jaime Sevilla / Epoch AI (byline: Denain & Wu per in-page citation)","date":"2026-03-23","note":"Adjacent finding (final training run vs. total R&D compute), not load-bearing for the core bridge question but relevant context on what 'training compute' means at a frontier lab. Fetched directly.","tier":1},{"url":"https://www.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2026/compute-power-ai.html","author":"Deloitte Insights (TMT Predictions 2026 team)","date":"2026","note":"Origin point of the widely-repeated 'two-thirds of AI compute is inference in 2026' figure already flagged unverified-quant on claim-inference-dominant-ai-compute-2026. Fetched directly to trace its citation.","tier":3},{"url":"https://www.weforum.org/stories/2025/01/scaling-ai-without-breaking-the-grid/","author":"Rodrigo Liang (CEO, SambaNova Systems)","date":"2025-01-03","note":"The root citation Deloitte's endnote points to for the two-thirds figure. Could not be fetched directly this session (403 Forbidden); traced via search-engine snippets instead, so recorded as [unsourced — needs verification] for its exact wording, but the authorship/venue/incentive facts (WEF contributor op-ed by an AI inference-chip vendor's CEO) are independently confirmed via search results and are sufficient on their own to disqualify it as a Tier 1-2 source for a quantitative claim.","tier":3},{"url":"https://telnyx.com/resources/ai-training-vs-inference","author":"Telnyx editorial","date":"2026","note":"The vault's existing source for claim-inference-dominant-ai-compute-2026's two-thirds figure — already flagged Tier 3 (audit V-002/V-011). Not re-fetched this session; referenced for continuity.","tier":3}]
related_notes: ["question-bridge-training-inference-compute","claim-inference-dominant-ai-compute-2026","claim-inference-cost-collapsed-280x","backpropagation-gap","claim-llm-inference-prefill-decode"]
---


Answering [[question-bridge-training-inference-compute]], which asked whether the compute asymmetry between [[training]] and [[inference]] is the load-bearing bridge missing between the vault's two largest clusters — [[backpropagation-gap|backpropagation-origins]] (~10 notes, spine claim [[backpropagation-gap]]) and the inference-economics cluster (~11 notes, spine claim [[claim-inference-dominant-ai-compute-2026]]).

**Short answer, established claim by claim below:** the *mechanism* half of the candidate bridge — training requires backpropagation's forward-and-backward pass, inference runs the forward pass alone, and that structural difference is what produces the compute/cost asymmetry — is now solidly Tier 1-sourced and does connect the two clusters. The *magnitude* half — the vault's current headline claim that inference is "roughly two-thirds" of all AI compute in 2026 — is not resolved by this research; if anything, tracing its citation chain back to its root makes it look weaker than before. The bridge note is real at the mechanism level and still ungrounded at the economic-dominance-statistic level. Both halves are recorded below so promotion sees exactly where the seam is solid and where it isn't.

---

## Claim 1: The FLOPs cost of a full training run and the FLOPs cost of a single inference differ by roughly twelve orders of magnitude for a concrete example model (GPT-3)

**Claim type:** quantitative
**Sourcing floor:** Tier 1–2 required; achieved Tier 1

> "For GPT-3, the cost of training was 3e23 FLOP, whereas the cost of a single inference is 3e11."

Pablo Villalobos and David Atkinson, Epoch AI, "Trading off compute in training and inference," 2023-07-28. 3e23 ÷ 3e11 = 1e12 — the training run is equivalent in FLOPs to roughly one trillion individual inferences.

**Provenance:** https://epoch.ai/blog/trading-off-compute-in-training-and-inference — Tier 1 (Epoch AI's own original compute-cost estimation, published on its own research venue, named authors).

---

## Claim 2: A commonly-used rule of thumb: single-inference compute is close to the square root of total training compute

**Claim type:** quantitative
**Sourcing floor:** Tier 1–2 required; achieved Tier 1

> "The single-inference compute is usually close to the square root of the training compute."

Same source and authors as Claim 1. The piece separately notes "significant variability" around this rule, so it is recorded as a heuristic, not a law.

**Provenance:** https://epoch.ai/blog/trading-off-compute-in-training-and-inference — Tier 1.

---

## Claim 3: The structural reason training is more expensive than inference, per token: training requires both a forward and a backward pass; inference requires only the forward pass; and the backward pass is roughly twice as expensive as the forward pass alone

**Claim type:** technical-mechanism
**Sourcing floor:** Tier 1–2 required; achieved Tier 1

> "When training a model, we need to perform both forward and backward passes, and backward passes are generally twice as expensive to do as forward passes."

> "tokens seen during training are around three times more expensive compared to tokens seen or generated during inference"

Ege Erdil, Epoch AI, "Optimally allocating compute between inference and training," 2024-03-29.

This is the precise mechanistic link the parent question asked for. The [[backpropagation-gap|backpropagation-origins cluster]] is, at root, about the algorithm that computes the backward pass — [[Seppo Linnainmaa|Linnainmaa]]'s reverse-mode automatic differentiation, later rediscovered as the neural-network training algorithm. The [[claim-inference-dominant-ai-compute-2026|inference-economics cluster]] is about the phase of a model's life that runs *without* that backward pass. Erdil's piece states directly, in FLOPs terms, why the second phase is cheaper than the first: the backward pass — backpropagation's own computational content — is the extra cost that only training pays.

**Provenance:** https://epoch.ai/blog/optimally-allocating-compute-between-inference-and-training — Tier 1 (Epoch AI's own derivation, named author, own venue).

---

## Claim 4: Under this framework, compute-optimal AI labs should in principle spend comparable total resources on training and on running inference, given flexible trade-offs between the two

**Claim type:** technical-mechanism / theoretical-economic claim (an allocation argument built from Claims 1–3, not an empirical measurement)
**Sourcing floor:** treated as Tier 1–2 given its mechanism-adjacent nature; achieved Tier 1

> "AI labs should spend comparable resources on training and running inference, assuming they can flexibly balance compute between these tasks to maintain model performance."

Same source as Claim 3. This is a normative/theoretical claim about optimal allocation, not a report of what labs actually spend (see Claim 5 for an empirical figure) — recorded as such, not conflated with observed spending.

**Provenance:** https://epoch.ai/blog/optimally-allocating-compute-between-inference-and-training — Tier 1.

---

## Claim 5: At one frontier lab in one year, actual compute spending was split roughly $3B training / $1.8B inference / ~$1–2B research, out of about $7B total cloud compute cost

**Claim type:** quantitative
**Sourcing floor:** Tier 1–2 required; achieved Tier 2, with an explicit caveat on the chain of custody

> "OpenAI spent $3 billion on training compute, $1.8 billion on inference compute, and $1 billion on research compute amortized over 'multiple years'."

Josh You, Epoch AI, "Most of OpenAI's 2024 compute went to experiments," 2025-10-10.

**Caveat, recorded rather than smoothed over:** the article itself states its dollar figures for OpenAI's overall spend come from "Epoch's AI Companies dataset, ultimately via reports in The Information and The New York Times" — i.e. Epoch is curating and modeling on top of numbers whose ultimate origin is journalism about a private company's unpublished finances, not OpenAI's own disclosure. Epoch's own methodology (stated assumptions, named analyst, transparent modeling) earns this a Tier 2 placement rather than the Tier 3 the raw press reports alone would carry — but it is not treated as a clean Tier 1 primary figure, and the number should not be read as more solid than "best current outside estimate."

**Provenance:** https://epoch.ai/data-insights/openai-compute-spend — Tier 2 (see caveat above).

---

## Claim 6 (re-audit of an existing vault claim): the "inference is roughly two-thirds of all AI compute in 2026, up from a third in 2023 and half in 2025" figure — already flagged `[unverified-quant — needs primary]` on [[claim-inference-dominant-ai-compute-2026]] — traces to a citation chain that does not clear the sourcing floor, and this session's research reinforces the existing flag rather than resolving it

**Claim type:** quantitative
**Sourcing floor:** Tier 1–2 required; **not achieved — remains below floor, now with the citation chain identified**

The identical "two-thirds / half / a third" figure appears verbatim or near-verbatim across multiple secondary venues (Telnyx, Vast.ai, Introl, and Deloitte). Tracing it to its earliest and most-cited instance:

> "Inference workloads will indeed be the hot new thing in 2026, accounting for roughly two-thirds of all compute (up from a third in 2023 and half in 2025)"

Deloitte Insights, "More compute for AI, not less" (2026 Technology, Media & Telecommunications Predictions), 2026. Tier 3 (named research team, real venue, but a predictions/trend report, not a primary compute measurement).

Deloitte's own endnote for this statistic cites: Rodrigo Liang, "Scaling AI without breaking the grid: The path to sustainable innovation," World Economic Forum, 2025-01-03. This WEF piece could not be fetched directly this session (403 Forbidden on direct request), so its exact wording is `[unsourced — needs verification]`; but its authorship is independently confirmed via search-engine results: Rodrigo Liang is the CEO of SambaNova Systems, a company that sells AI inference chips. A WEF contributor op-ed by a vendor CEO with a direct commercial stake in "inference is the dominant, growing workload" being true is a Tier 3-at-best source for this specific statistic under the sourcing floor's own logic (named human, real-ish venue, but an incentive-laden opinion platform, not a measurement or dataset) — and it is one link further from any actual compute-measurement methodology than even Deloitte's own report.

**Conclusion for this claim:** the "two-thirds" figure that anchors [[claim-inference-dominant-ai-compute-2026]]'s framing has now been traced past its most commonly-cited restatement (Deloitte) to its underlying citation (a vendor CEO's advocacy op-ed), and at no point in that chain does a primary compute-measurement source appear. This does not overturn the existing `[unverified-quant — needs primary]` flag on that claim-note — it corroborates it with a specific reason to distrust the number rather than merely an absence of a better one. Recorded here as `[unverified-quant — needs primary]`, same as the original note, with the new citation-chain evidence attached.

**Provenance:** https://www.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2026/compute-power-ai.html — Tier 3. Underlying citation: https://www.weforum.org/stories/2025/01/scaling-ai-without-breaking-the-grid/ — Tier 3, wording `[unsourced — needs verification]`, authorship/incentive facts independently confirmed.

---

## Answering the core question

The candidate bridge claim in [[question-bridge-training-inference-compute]] had two parts: (a) a mechanism claim — training is backpropagation's domain (forward + backward pass), inference is the forward pass alone — and (b) an implicit magnitude claim — that this asymmetry is what makes inference "the dominant AI compute workload" today, giving [[claim-inference-dominant-ai-compute-2026]] the grounding it currently lacks.

Part (a) is now confirmed at Tier 1, cleanly, via Epoch AI's own compute-economics research (Claims 1–4 above). This is a genuine, load-bearing, correctly-sourced bridge between the two clusters: the algorithm at the center of the backpropagation-origins cluster (reverse-mode differentiation / the backward pass) is precisely the extra computational step that training pays and inference does not, and that omission is the mechanistic reason inference is cheaper per unit — a fact independently derivable from first principles (Erdil's forward/backward-pass accounting) and empirically illustrated (Villalobos & Atkinson's GPT-3 FLOPs example).

Part (b) is not confirmed by this research, and the attempt to confirm it surfaced a specific, checkable reason to doubt it: the vault's currently-cited "two-thirds of compute" figure traces to a chip-vendor CEO's advocacy op-ed, not a measurement. What Claim 5 offers instead is a real, if narrower and Tier-2, empirical data point (one lab, one year, roughly even split between training and inference spend, with research/experiments actually the largest single category) — a figure that is consistent with Claim 4's theoretical "comparable spending" prediction but does not itself establish an industry-wide "inference now dominates" trend.

**Recommended framing for the eventual claim-note this capture is meant to seed:** write the bridge as the mechanism claim (a), sourced to Claims 1–4, and treat it as fully separable from the magnitude claim in [[claim-inference-dominant-ai-compute-2026]], which should keep its own `[unverified-quant]` flag rather than borrowing false confidence from the newly-sourced mechanism. The two clusters can be linked correctly without over-claiming the economic-dominance framing that motivated the original question.

---

## Further leads

- The still-dangling [[claim-inference-engineering-emerged-as-specialty]] (referenced by three notes, never written) sits on the same seam as this bridge; deciding write-or-unlink was explicitly deferred to "when the bridge note lands" — this capture is that landing point, so the decision is now unblocked.
- Epoch AI's "final training run vs. total R&D compute" piece (Denain & Wu, 2026-03-23, https://epoch.ai/gradient-updates/r-and-d-vs-training-compute) is a Tier 1 source not yet used anywhere in the vault; it complicates "training compute" itself (only ~10% of a frontier lab's R&D compute goes to the released model's final run) and could sharpen [[claim-inference-dominant-ai-compute-2026]]'s framing once that note is re-sourced.
- The WEF op-ed (Rodrigo Liang, 2025-01-03) could not be fetched directly (403). Worth a retry via an archived copy (Wayback Machine) to confirm or refute whether it actually contains the "two-thirds" figure at all, since even Deloitte's own citation of it as the source for that number could not be verified word-for-word this session.
- No industry-wide, Tier 1–2, primary-measurement source for "share of global AI compute spent on inference vs. training" was found in this session despite a direct search for one. This looks like a genuine gap in the public record, not a search failure — worth logging as its own open question (a companion to [[claim-inference-dominant-ai-compute-2026]]'s existing flag) rather than assuming the next search will find it.

> [!note] Seek's commentary:
> The interesting result here is a split verdict, and the split is itself informative. The vault's instinct that these two clusters share a seam was right — backpropagation's backward pass is not a vague thematic echo of inference economics, it is the literal FLOPs term whose absence is being priced. But the number that made the bridge feel *urgent* (two-thirds! inference is eating the industry!) turned out, on tracing, to run through a chip-vendor's advocacy piece rather than a measurement. That's worth sitting with: the mechanism-level bridge and the magnitude-level headline arrived from completely different evidentiary places, and conflating them — writing one bridge note that leans on both — would quietly launder the weak one's credibility through the strong one's. Better to write two notes, or one note with an unambiguous internal seam, than one smooth-sounding bridge.