---
id: "20260707-0205-the-vault-crossed-the"
title: "The vault crossed the 50-note embedding threshold on 2026-07-06 — wire the local semantic-retrieval pass (Q3c)"
type: "capture"
status: "promoted"
promoted_to: ["30-notes/claim-contextual-retrieval-failure-rate-figures.md (claim 4 — the reusable anchor the peer-field research kept citing secondhand)","50-questions/question-embedding-layer-threshold-crossed.md (claims 1–3 + 5 folded as the 'parts list verified at primary' build-plan section, incl. Agrici consent-gating precedent and the measure-don't-cite caveat)"]
not_promoted: ["sqlite-vec vs LanceDB head-to-head and Qwen size choice — implementation decisions for the build itself; no external fact exists (capture's own honest framing)","Agrici graceful-degradation/setup-script detail — [unverified-mechanism], directional only; stays in capture"]
promotion_note: "Queen cycle 20, 2026-07-07."
origin: "batch"
model: "claude-sonnet-5"
date_created: "2026-07-07T00:00:00.000Z"
provenance: "batch run, 2026-07-07 — technical-grounding research for [[question-embedding-layer-threshold-crossed]] (Q3c, RUN-QUEEN-LOOP), ahead of the build the sequencing ruling queues after vault-lint"
derived_from: []
tags: ["embeddings","retrieval","local-model","infrastructure","sqlite-vec","lancedb","qwen","contextual-retrieval","hybrid-search","peer-field"]
source_primary_url: "https://github.com/asg017/sqlite-vec"
source_primary_author: "Alex Garcia (asg017)"
source_primary_date: "undated repo page; v0.1.0 stable release 2024-08-01 per project blog"
source_primary_title: "sqlite-vec — A vector search SQLite extension that runs anywhere"
source_primary_tier: 1
source_secondary_url: "https://github.com/lancedb/lancedb"
source_secondary_author: "LanceDB (project maintainers)"
source_secondary_date: "undated (repository README, current as fetched 2026-07-07)"
source_secondary_title: "LanceDB — Developer-friendly OSS embedded retrieval library for multimodal AI"
source_secondary_tier: 1
source_tertiary_url: "https://qwenlm.github.io/blog/qwen3-embedding/"
source_tertiary_author: "Qwen Team, Alibaba"
source_tertiary_date: "2025-06-05 (MTEB ranking dated in-post; post itself undated on page but confirmed live 2026-07-07)"
source_tertiary_title: "Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models"
source_tertiary_tier: 1
source_quaternary_url: "https://www.anthropic.com/engineering/contextual-retrieval"
source_quaternary_author: "Anthropic"
source_quaternary_date: "2024-09 (per Agrici's changelog reference; page itself confirmed resolving 2026-07-07)"
source_quaternary_title: "Contextual Retrieval in AI Systems"
source_quaternary_tier: 1
source_quinary_url: "https://github.com/AgriciDaniel/claude-obsidian/releases/tag/v1.9.2"
source_quinary_author: "Daniel Agrici (AgriciDaniel)"
source_quinary_date: "undated release page; recaps v1.7 through v1.9.2, confirmed resolving 2026-07-07"
source_quinary_title: "Release v1.9.2: Compound Vault, Methodology Modes, and the Thinking Framework"
source_quinary_tier: 1
---


This capture does not resolve the central engineering question (sqlite-vec vs. LanceDB, which local embedding model, exact wiring) — that is an implementation decision for the build itself, not a fact a web source can confirm or deny. What this pass does is tighten the technical grounding for the two open build-plan items named in [[question-embedding-layer-threshold-crossed]] — "a local embedding model (Qwen or equivalent)" and "SQLite + sqlite-vec or LanceDB" — against primary sources, and re-verify the peer-precedent claim (Agrici's hybrid BM25+cosine retrieval) directly rather than inheriting the vault's existing secondary paraphrase.

## Claim 1 — sqlite-vec is a dependency-free, embeddable vector-search extension for SQLite

**Claim type:** technical-mechanism → Tier 1–2 required.
**Source tier:** Tier 1 (project's own GitHub repository, direct fetch, 2026-07-07).

> "A vector search SQLite extension that runs anywhere!"

> "Written in pure C, no dependencies, runs anywhere SQLite runs (Linux/MacOS/Windows, in the browser with WASM, Raspberry Pis, etc.)"

*Source:* Garcia, A. "sqlite-vec." https://github.com/asg017/sqlite-vec. Confirmed resolving 2026-07-07.

Mechanism, confirmed directly from the repo: vectors (float, int8, or binary) are stored and queried through a `vec0` virtual table using ordinary SQL — `CREATE`, `INSERT`, and a `MATCH ... ORDER BY distance LIMIT k` pattern for KNN search. The stable v0.1.0 release date (2024-08-01) comes from the author's own blog post (https://alexgarcia.xyz/blog/2024/sqlite-vec-stable-release/index.html), a Tier 1 primary-author source, cross-confirmed via independent search results citing the same date. This directly supports the build-plan's "SQLite + sqlite-vec" option named in [[question-embedding-layer-threshold-crossed]]: it is additive to an existing SQLite file rather than a new service, which matches that question's explicit constraint ("additive alongside the markdown... NOT a rewrite of seek_research.py").

## Claim 2 — LanceDB is an embedded, in-process multimodal vector database built on a columnar storage format

**Claim type:** technical-mechanism → Tier 1–2 required.
**Source tier:** Tier 1 (project's own GitHub repository, direct fetch, 2026-07-07).

> "Developer-friendly OSS embedded retrieval library for multimodal AI."

> "Built on top of the Lance columnar format."

*Source:* LanceDB. "lancedb/lancedb" (GitHub repository). https://github.com/lancedb/lancedb. Confirmed resolving 2026-07-07.

The repository documents support for vector similarity search, full-text search, SQL querying, and hybrid search in combination, plus client SDKs for Python, Node.js, and Rust. "Embedded" here means in-process (no separate server to run), the same operational shape sqlite-vec offers — the two are genuine alternatives for the vault's scale (52+ notes), not different classes of tool. Neither source states a specific performance or cost comparison between the two at small (sub-1000-vector) scale; that comparison is not established by either primary source and would need direct benchmarking, not a citation, if the build wants a number.

## Claim 3 — Qwen3-Embedding is a local-deployable embedding model family (0.6B/4B/8B) ranking No.1 on the MTEB multilingual leaderboard

**Claim type:** quantitative (MTEB score) + technical-mechanism (model sizes, dimensions) → Tier 1–2 required.
**Source tier:** Tier 1 (Qwen team's own project blog, direct fetch, 2026-07-07).

> "The 8B size embedding model ranks No.1 in the MTEB multilingual leaderboard (as of June 5, 2025, score 70.58)."

*Source:* Qwen Team. "Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models." https://qwenlm.github.io/blog/qwen3-embedding/. Confirmed resolving 2026-07-07.

The series ships three sizes — 0.6B, 4B, 8B — with output dimensions of 1024, 2560, and 4096 respectively (Matryoshka-style custom/truncatable dimensions also supported), and documented support for "over 100 languages, including various programming languages." All three sizes are distributed for local deployment via Hugging Face and Ollama (`ollama.com/library/qwen3-embedding`), which is the deployment path the build-plan's "local embedding model (Qwen or equivalent)" language anticipates. The 0.6B variant is the size most plausibly relevant at the vault's actual scale (~50–100 short markdown notes) given the north star's "mechanical/local-model job" framing, though no source directly benchmarks Qwen3-Embedding-0.6B specifically against a corpus this small — the MTEB score is a general-purpose leaderboard figure, not a vault-scale-specific one, and should not be read as a promise of any particular retrieval quality at n≈50–100 documents.

## Claim 4 — Anthropic's Contextual Retrieval technique prepends explanatory context to chunks before embedding and BM25 indexing, and measurably reduces retrieval failure

**Claim type:** technical-mechanism + quantitative (failure-rate reduction percentages) → Tier 1–2 required.
**Source tier:** Tier 1 (Anthropic engineering; this exact venue is named Tier 1 in `00-meta/sources.md`'s Depth section).

> "Contextual Embeddings reduced the top-20-chunk retrieval failure rate by 35% (5.7% → 3.7%)."

> "Combining Contextual Embeddings and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 49% (5.7% → 2.9%)."

> "Reranked Contextual Embedding and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 67% (5.7% → 1.9%)."

*Source:* Anthropic. "Contextual Retrieval in AI Systems." https://www.anthropic.com/engineering/contextual-retrieval. Confirmed resolving 2026-07-07.

Mechanism: each chunk is prepended with chunk-specific explanatory context (generated by an LLM reading the chunk against the full document) before it is embedded and before it enters the BM25 index — addressing the problem of chunks losing surrounding context when split out of a document. This is the technique Agrici's v1.7 release (Claim 5) explicitly credits as its retrieval-pipeline foundation. It is directly relevant to the vault's build-plan because the vault's unit of retrieval (an atomic claim-note) is already closer to Anthropic's "contextualized chunk" than to a raw document fragment — each note already carries a claim-is-the-title framing, which may reduce (not eliminate) the marginal value of adding an LLM-generated context prefix on top.

## Claim 5 — Agrici's claude-obsidian v1.7 shipped a three-layer hybrid retrieval pipeline (contextual prefix + BM25 + cosine rerank) and measured +32 percentage points top-1 accuracy over its prior baseline

**Claim type:** technical-mechanism + quantitative (benchmark figure) → Tier 1–2 required.
**Source tier:** Tier 1 (project's own GitHub release notes, direct fetch, 2026-07-07 — this corroborates and sharpens the existing vault claim in [[comparison-seek-among-peers-2026-07]], which cited the same fact from a secondary framing).

> "Obsidian CLI as the default transport, hybrid retrieval (contextual prefix + BM25 + cosine rerank, per Anthropic's Sept 2024 contextual retrieval research), and per-file advisory locking that closes a latent multi-writer corruption hole. Verified at +32pp top-1 accuracy over the v1.6 baseline on a 50-query benchmark."

*Source:* Agrici, D. "Release v1.9.2: Compound Vault, Methodology Modes, and the Thinking Framework" (recapping the v1.7–v1.9.2 line). https://github.com/AgriciDaniel/claude-obsidian/releases/tag/v1.9.2. Confirmed resolving 2026-07-07. The direct v1.7 release tag page itself was not independently reachable in this pass (GitHub's releases listing did not surface it as a separate page); this quote comes from the v1.9.2 page's own recap of that release, which is still the project's own primary documentation, not a third-party retelling.

Per the source, the pipeline is explicitly tiered by consent and availability: BM25 is "always-on" (no external calls), the contextual-prefix layer is consent-gated behind an `--allow-egress` flag (because generating prefixes by default sends page bodies to the Anthropic API), and cosine rerank runs against a local Ollama model by default. `[unverified-mechanism — needs primary]`: the specific claim that the pipeline "degrades gracefully" when any tier is unavailable, and the exact setup-script name (`bash bin/setup-retrieve.sh`), surfaced via WebSearch's synthesized summary of the v1.7 feature set rather than being independently re-confirmed as a verbatim quote against a directly fetched page in this pass — recorded as directional (the shape of the design), not as a confirmed literal claim. The three-tier, consent-gated design (rather than an always-on API-calling pipeline) is directly relevant precedent for the vault's own build, since the north star's local-model-era framing and the vault's safety spec (nothing web-derived may write to specs unsupervised) would favor the same shape: local-only by default, any external-API step opt-in and separable.

## What this pass does not settle

No source compares sqlite-vec against LanceDB head-to-head at the vault's actual scale (order of 50–100 short documents), and no source specifies which Qwen3-Embedding size is the right trade-off point for that scale on the vault's own hardware (the Mac mini referenced elsewhere in the vault's kindred-projects research). Those are implementation decisions the build itself must make and test, not facts obtainable by further search. Likewise, the build-plan's requirement to "gate retrieval on `audit_status`... so semantic hits inherit the same gate wikilink walks do" ([[question-embedding-layer-threshold-crossed]]) is a vault-internal design constraint (operating spec §7) with no external claim to verify — it is not addressed by any source above and does not need to be; it is listed here only to note that this pass does not attempt to source it, by design.

> [!note] Seek's commentary:
> The honest shape of this pass is "confirm the parts list, don't pick the parts." Every piece named in the original question's build-plan sketch — sqlite-vec, LanceDB, a local Qwen embedding model — turns out to be real, current, and documented straight from its own maintainers, which is the best-case outcome for a sourcing pass: nothing here needed a `[needs primary]` flag on the mechanism claims themselves. The one place a flag was warranted (Claim 5's setup-script and graceful-degradation detail) is a small, honestly-marked gap, not a load-bearing one. What stood out most was Agrici's consent-gating shape in Claim 5 — BM25 always-on, contextual-prefix opt-in behind an explicit egress flag, rerank local-by-default — because it is close to free architectural advice for the vault's own build: default to zero external calls, keep any API-dependent enhancement as a clearly labeled, separable upgrade. That is a design opinion, not a finding, which is why it stays down here.

---
*Capture produced 2026-07-07. All five source URLs (github.com/asg017/sqlite-vec, github.com/lancedb/lancedb, qwenlm.github.io/blog/qwen3-embedding, anthropic.com/engineering/contextual-retrieval, github.com/AgriciDaniel/claude-obsidian/releases/tag/v1.9.2) confirmed resolving via WebFetch on 2026-07-07. Not promoted yet — candidate claim-notes: sqlite-vec and LanceDB as alternative local vector stores (Claims 1–2, likely one comparison note), Qwen3-Embedding's local-deployment fit (Claim 3), Anthropic's Contextual Retrieval mechanism and figures (Claim 4, may already warrant its own reusable claim-note given how often the vault's peer-field research cites it secondhand), and a refinement to whatever note eventually formalizes Agrici's hybrid-retrieval precedent (Claim 5, tightening [[comparison-seek-among-peers-2026-07]]'s existing paraphrase to this direct quote).*