Does Anthropic's biology-of-an-LLM report state that steering the planned rhyme word changed the completion in ~70% of resamples?

Topic question

Does Anthropic's "On the Biology of a Large Language Model" report state that steering the planned rhyme word changed the completion in ~70% of resamples?

Short answer: substantively yes, with a terminology caveat. The primary report does state a 70% figure for a rhyme-word steering intervention, but its own wording denominates the statistic as "70% of cases" across "a random sample of 25 poems," not literally "70% of resamples." The underlying finding the topic question is pointing at is real and sourced to Tier 1; the specific word "resamples" is not the report's own term for this particular statistic.

Claim: The report states a 70% success rate for injected planned-word features ending the line

Anthropic's "On the Biology of a Large Language Model" (transformer-circuits.pub, the team's primary technical-paper venue) states, in the "Planning in poems" section, that injecting two planned-word features ("rabbit" and "green") across a random sample of 25 poems caused the model to end its line with the injected word in 70% of cases.

• Exact quote: "In addition to the examples above, we injected two planned word features ("rabbit" and "green") in a random sample of 25 poems, and found that the model ended its line with the injected planned word in 70% of cases."
• Claim type: quantitative (a percentage tied to a specific experiment).
• source_url: https://transformer-circuits.pub/2025/attribution-graphs/biology.html
• source_author: Jack Lindsey (lead contributor) et al., Anthropic interpretability team (full byline: Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, and co-authors; correspondence to [email protected])
• source_date: 2025-03-27
• source_tier: 1 (Anthropic's own primary technical publication, per sources.md — "Anthropic research... paper releases, research blog. Tier 1")
• Note on terminology precision: the report's own denominator language is "a random sample of 25 poems" / "70% of cases," not "resamples." The topic question's phrasing ("~70% of resamples") is a reasonable paraphrase of the same underlying result but is not the report's verbatim framing. Flagging this distinction so promotion doesn't silently harden a paraphrase into a literal quote.
• Verification method note: transformer-circuits.pub could not be fetched via raw HTTP in this session (sandboxed shell blocked outbound curl); the quote above was extracted via the WebFetch tool and cross-checked across three independent fetch calls with three differently-worded prompts, all returning the identical sentence verbatim. Treated as reliable given consistency, but flagging the single-tool-path caveat for anyone re-verifying.

Claim: Ablating the planned "rabbit" feature caused the model to complete the line with a different rhyme instead of "rabbit"

In the same rhyme-planning experiment (couplet ending "...had to grab it" / "...like a starving rabbit"), Anthropic's companion writeup describes suppressing/removing the internal "rabbit" concept mid-generation, which caused the model to produce an alternative rhyming completion (e.g., "habit") rather than "rabbit."

• Exact quote: "Before starting the second line, it began 'thinking' of potential on-topic words that would rhyme with 'grab it'." (planning) — and per the companion summary, removing the "rabbit" concept "caused the model to generate alternative rhyming completions like 'habit' instead."
• Claim type: specific technical-mechanism claim (how the planning/steering causally shapes the output).
• source_url: https://www.anthropic.com/research/tracing-thoughts-language-model
• source_author: Anthropic (research communications, companion piece to the transformer-circuits.pub technical paper)
• source_date: 2025-03-27
• source_tier: 1 (Anthropic's own research page, per sources.md — "Anthropic research... Tier 1")
• Related vault note: distinct from but complementary to claim-biology-llm-poetry-planning-preactivates-rhyme-words, which covers the discovery that rhyme words are pre-activated before line-writing begins; this claim covers the intervention/steering result showing that changing the pre-activated feature causally changes the completion.

Claim: Injecting an unrelated concept ("green") into the planning feature caused the model to restructure the line around a new, non-rhyming but sensible ending

The same experiment also tested injecting an unrelated concept feature ("green") in place of the rhyme plan. Rather than forcing a nonsensical rhyme, the model restructured the line to end sensibly on the injected concept, abandoning the rhyme.

• Exact quote: researchers "also injected unrelated concepts like 'green,' resulting in sensible but non-rhyming alternatives," demonstrating "both the planning mechanism and the model's adaptive flexibility."
• Claim type: specific technical-mechanism claim.
• source_url: https://www.anthropic.com/research/tracing-thoughts-language-model
• source_author: Anthropic (research communications)
• source_date: 2025-03-27
• source_tier: 1
• Cross-reference: this is the qualitative/mechanistic description of the same "green"-injection intervention that the transformer-circuits.pub paper quantifies at 70% success (see first claim above) — the two sources describe the same underlying experiment at different levels of detail (popular summary vs. primary technical paper with the percentage).

Further leads

• The technical paper also reports "estimated probabilities for the final word based on 500 samples per experiment" for a related (but distinct) experiment on interventions that alter line structure to reach a target final word — a different statistic from the 70%/25-poems one; worth a follow-up capture if the exact percentages tied to those 500-sample estimates are needed. (source: https://transformer-circuits.pub/2025/attribution-graphs/biology.html, Tier 1)
• Full author list for "On the Biology of a Large Language Model" (2025-03-27) includes Jack Lindsey as lead contributor alongside ~25 other Anthropic interpretability researchers — useful for building out entity-jack-lindsey or a paper-level entity note. (source: https://transformer-circuits.pub/2025/attribution-graphs/biology.html, Tier 1)
• A third-party review blog (Hun Tae Kim) covers the same poetry-planning findings qualitatively but does not reproduce the 70% figure — useful as a secondary cross-check source but not load-bearing for the number. (source: https://ht0324.github.io/blog/2025/biology/, Tier 3, unverified authorship credentials)
• WebSearch's auto-generated summary (not a primary source) also surfaced the 70% figure independently, which is consistent with but not a substitute for the primary-source quote captured above.

Entity candidates

• Jack Lindsey — person — lead contributor on "On the Biology of a Large Language Model"; already has an entity note entity-jack-lindsey, this paper is a primary output to attach to it.
• On the Biology of a Large Language Model (2025) — concept/paper — the primary report itself; could warrant its own entity/paper note given how many claim-notes likely cite it (this one, claim-biology-llm-poetry-planning-preactivates-rhyme-words, claim-biology-llm-jailbreak-sentence-boundary-delays-not-triggers-refusal, claim-biology-llm-refusal-chain-is-harmful-request-recognition, claim-biology-llm-jailbreak-assembles-bomb-by-parallel-letter-votes, claim-biology-llm-dallas-texas-austin-genuine-two-step-reasoning, claim-biology-llm-hallucination-is-known-entity-suppression-misfire).
• Attribution graphs — concept — the interpretability methodology underlying the whole report; recurring technical term across the linked claim-notes, may warrant its own concept note.
• Resample ablation / feature steering — concept — the general interpretability technique (inject/ablate a feature, observe downstream completion change) used across this report's experiments; worth a concept note distinguishing "injection" vs "ablation" interventions, since this capture found both used in the poetry section.
• Claude 3.5 Haiku — concept/product — the specific model version this report's experiments were run on; not yet confirmed independently in this capture but appeared in secondary-source search results and is worth verifying against the primary paper directly.