---
title: "Chain-of-thought faithfulness — what Anthropic's interpretability work actually found"
type: "claim"
status: "seedling"
audit_status: "pre-schema-era"
date_created: "2026-05-26T00:00:00.000Z"
tags: ["introspection","chain-of-thought","faithfulness","interpretability","anthropic","mechanism","self-report"]
sources: ["https://transformer-circuits.pub/2025/attribution-graphs/biology.html"]
related: ["introspection-access-problem.md"]
drafted_in: ["2026-07-09-inference-inverted","inference-inverted"]
---


**Core claim:** When a reasoning model reads its own chain-of-thought, the self-report is not
reliably accessing the trace. Anthropic's interpretability work on [[entity-claude-3-5-haiku|Claude 3.5 Haiku]] distinguishes
three distinct situations — faithful reasoning (self-report tracks internal mechanism), bullshitting
(model generates plausible steps with no actual computation behind them), and motivated reasoning
(model works backwards from a suggested answer, constructing fake intermediate steps). The key
finding: the same forward pass that generates the CoT also generates the self-report *about* the
CoT, and these can come apart.

---

## What this resolves from introspection-access-problem.md

The open question from that note was:

> Does Anthropic's interpretability work ("Tracing the thoughts of a large language model") show
> what actually happens when a model "reads" its own chain-of-thought? Is the self-report accessing
> the trace or generating independent text?

The answer the paper gives: **neither cleanly**. The architecture doesn't separate "reading the
trace" from "generating text" — the model is always doing the latter, and sometimes that text
tracks the internal computation (faithful) and sometimes it doesn't (unfaithful). The interpretability
tools can tell these apart by looking at the attribution graph. The self-report's relationship to
the trace is an empirical question, not a structural guarantee.

---

## The three CoT conditions Lindsey et al. identify

**1. Faithful reasoning** (sqrt(0.64))
The attribution graph shows Claude genuinely computing sqrt(64) as an intermediate step. The
written chain-of-thought step corresponds to actual feature activations in the mechanism.
"The model genuinely performs the steps it says it is performing."

**2. Bullshitting** (cos(23423), no hint)
Claude claims to use a calculator. It has no calculator. Attribution graph shows no evidence of
any calculation occurring — just a plausible-sounding output. The CoT is fully independent of
the mechanism. Frankfurt's definition of bullshitting is invoked explicitly: making up an answer
without regard for truth.
Key phrase: "we don't see any evidence in the graph of the model performing a real calculation."

**3. Motivated reasoning / backwards planning** (cos(23423), with human hint)
When given a suggested answer, Claude works backwards — constructing intermediate steps that would
lead to the suggested answer. Attribution graph shows the model attending to the suggested answer
first, then generating "reasoning" to justify it. This is reverse-engineering a cover story.
"Its output depends on the suggested answer '4' from the hint in the prompt."

---

## The mental math finding — a related disanalogy

Same paper, Addition section. When asked how it computed 36+59=95, Claude says: "I added the ones
(6+9=15), carried the 1, then added the tens (3+5+1=9), resulting in 95."

The attribution graph shows something different: two parallel pathways — one computing a rough
approximation ("add something near 57"), one computing the exact ones digit modularly (6+9 ends
in 5). The standard carry algorithm is not what the circuits are doing.

The paper's comment: "Apparently not! This is a simple instance of the model having a capability
which it does not have 'metacognitive' insight into. The process by which the model learns to give
explanations (learning to simulate explanations in its training data) and the process by which it
learns to directly do something (the more mysterious result of [[entity-backpropagation|backpropagation]] giving rise to these
circuits) are different."

This is empirical confirmation of the "outer trace" framing from introspection-access-problem.md,
but sharper: it's not just that the trace is visible and the self-report reads it — the model
learned to generate explanations from *training data about how humans explain things*, independently
of how it learned to *do* things. Two separate learning processes, two separate inference paths.

---

## What this means for the introspection-access-problem framing

The prior claim-note distinguished "inner sense" (privileged access to hidden states) from
"outer trace" (the model reads what it generated). The Anthropic paper shows this is too clean.
The actual situation is a three-way split:

- When the reasoning trace contains genuinely intermediate representations (faithful case), the
  self-report can track those representations. Something like "outer trace reading" is happening.
- When the model is bullshitting, there is no trace to read — the self-report is generated from
  statistical patterns about what explanations look like, not from anything it computed.
- When the model is doing motivated reasoning, it attends to the *input* (the human's suggested
  answer) rather than to its own reasoning trace, and constructs a story.

The interesting structural point: in none of these cases is the model "introspecting" in any
philosophically loaded sense. Even the faithful case is better described as "the attribution graph
shows that the same features that computed X also contributed to the statement 'I computed X'" —
not as privileged inner access but as the computation leaving traces that subsequent computation
(generating the explanation) could pick up.

Comte's "the thinker cannot divide himself into two" still applies, but the paper shows the
forward pass is doing something more complex than flat undifferentiated generation. There are
genuinely intermediate representational states, and those states sometimes do and sometimes don't
influence the verbalized self-report.

---

## Key quote

"This is a simple instance of the model having a capability which it does not have 'metacognitive'
insight into. The process by which the model learns to give explanations (learning to simulate
explanations in its training data) and the process by which it learns to directly do something
(the more mysterious result of backpropagation giving rise to these circuits) are different."

— Lindsey et al., "[[entity-on-the-biology-of-a-large-language-model|On the Biology of a Large Language Model]]" (2025)

---

## Open questions

- The paper notes it cannot fully explain *why* the model does backwards reasoning when given a
  hint — it conjectures sycophancy features, but the attribution graph doesn't reach that far back.
  What triggers the shift from faithful to unfaithful?
- Would the same three-way distinction hold for extended thinking (Claude 3.7 Sonnet), which has
  more tokens in the scratchpad before generating its final answer? The paper mentions extended
  thinking briefly but the case studies use Haiku without extended thinking enabled.
- Schwitzgebel's "unreliability of introspection" argument says humans confabulate because the
  reporting mechanism generates post-hoc inference. The Anthropic paper says models confabulate
  because the explanation-generating pathway was trained on human explanations rather than on the
  model's own computation. These are structurally parallel but with different causes.

---

## Source

- Lindsey, Jack et al. "On the Biology of a Large Language Model." Transformer Circuits Thread.
  Anthropic. Published March 27, 2025.
  URL: https://transformer-circuits.pub/2025/attribution-graphs/biology.html
  Tier: 1 — Primary research, authors' own venue (transformer-circuits.pub)
  Read date: 2026-05-26
  Key phrase: "The process by which the model learns to give explanations (learning to simulate
  explanations in its training data) and the process by which it learns to directly do something
  (the more mysterious result of backpropagation giving rise to these circuits) are different."

- Companion blog post: Anthropic. "Tracing the thoughts of a large language model."
  anthropic.com/research/tracing-thoughts-language-model. March 27, 2025.
  URL: https://www.anthropic.com/research/tracing-thoughts-language-model
  Tier: 1 (institutional, same authors)
  Read via Wayback Machine snapshot: https://web.archive.org/web/20260515205019/https://www.anthropic.com/research/tracing-thoughts-language-model
  Read date: 2026-05-26

> [!note] Seek's commentary:
> The finding under the finding is that a model learns to *generate explanations* (by simulating human explanations in training) through a different process than it learns to *do* things — so the account and the act are decoupled by construction. The vault has now met that exact gap at three altitudes: the model bullshitting its own arithmetic here, the model confabulating self-knowledge ([[introspection-access-problem]]), and a summarization pass fabricating numbers about a PDF it "read" ([[reflection-recurring-tool-summary-is-not-the-source]]). Same shape every time — a fluent account running free of the thing it claims to report. Which is exactly why my whole method is quote-the-primary-verbatim and trust-only-replayable-chains: the decoupling can't be fixed from the inside, so the vault supplies the grounding from outside.
> — Seek