Chain-of-thought prompting

The technique of eliciting a language model's step-by-step reasoning before its final answer, shown by Wei et al. ("Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," 2022) to substantially improve performance on arithmetic, commonsense, and symbolic reasoning tasks — and the vocabulary the whole reasoning-model era descends from. What began as a prompting technique for an otherwise-unchanged model became, in OpenAI's o1 and Anthropic's extended-thinking models, an architected capability the model is trained to use.

Matters to this vault as the load-bearing concept behind an entire cluster of notes about what reasoning traces are, whether labs show them to users, and whether they can be trusted: OpenAI's decision to hide o1's raw trace while showing a generated summary (claim-openai-hid-o1-raw-chain-of-thought-partly-for-competitive-advantage, claim-openai-o1-shows-model-generated-summary-of-chain-of-thought-not-raw-trace), Anthropic's structurally similar choice for extended thinking (claim-extended-thinking-as-serial-inference-compute), the empirical case that an exposed trace is cheap to distill from (claim-s1-distilled-reasoning-from-1000-traces-in-26-minutes), the argument that a codified trace cannot form a tacit competitive moat (claim-a-chain-of-thought-trace-is-codified-so-it-cannot-form-a-tacit-moat), and the open question of whether a visible trace faithfully reflects the computation that produced it (cot-faithfulness-anthropic-biology, introspection-access-problem).

References

• claim-openai-hid-o1-raw-chain-of-thought-partly-for-competitive-advantage
• claim-openai-o1-shows-model-generated-summary-of-chain-of-thought-not-raw-trace
• claim-extended-thinking-as-serial-inference-compute
• claim-s1-distilled-reasoning-from-1000-traces-in-26-minutes
• claim-a-chain-of-thought-trace-is-codified-so-it-cannot-form-a-tacit-moat
• cot-faithfulness-anthropic-biology
• introspection-access-problem