Anthropic's Contextual Retrieval — prepending chunk context before embedding and BM25 cuts top-20 retrieval failure by 35–67%
Mechanism (Tier 1, direct fetch): each chunk gets an LLM-generated, chunk-specific explanatory prefix (written by reading the chunk against its full document) before embedding and before BM25 indexing — attacking the problem that chunks lose their surrounding context when split out. The ladder of verified figures:
- Contextual Embeddings alone: top-20 failure 5.7% → 3.7% (−35%)
-
- Contextual BM25: → 2.9% (−49%)
-
- reranking: → 1.9% (−67%)
Why the vault keeps this. (1) It is the acknowledged foundation of the nearest peer precedent — Agrici's claude-obsidian v1.7 hybrid pipeline ("contextual prefix + BM25 + cosine rerank, per Anthropic's Sept 2024 contextual retrieval research," +32pp top-1 on a 50-query benchmark, project's own release notes) — see comparison-seek-among-peers-2026-07. (2) For the vault's own Q3c build (question-embedding-layer-threshold-crossed): an atomic claim-note is already closer to a contextualized chunk than a raw fragment — the claim IS the title, provenance rides in frontmatter — so the marginal value of adding an LLM prefix on top is plausibly lower here than in Anthropic's document-splitting setting. That's a hypothesis the build can test cheaply, not a reason to skip the BM25+rerank layers, whose gains are separate. moc-peer-field-agent-memory.
Source
“Reranked Contextual Embedding and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 67% (5.7% → 1.9%).”