question open 2026-07-11

Should the vault's single source_tier field split into two axes — source reliability separate from claim credibility — the way intelligence doctrine and RAG both do?

vault-design source-tiers provenance epistemics meta

The Admiralty-Code hop (observation-intelligence-doctrine-and-rag-independently-derived-a-two-axis-source-model) surfaces a direct challenge to the vault's own design. Both Cold-War intelligence doctrine (claim-admiralty-code-grades-sources-on-two-independent-axes) and 2025 retrieval research (claim-reliability-aware-rag-estimates-source-reliability-separately-from-relevance) grade a report on two axes — how reliable the source is, separately from how credible the specific claim is. The vault's source_tier fuses these into one number.

The question. Should a claim-note carry two fields — e.g. a source-reliability tier and a per-claim credibility/corroboration grade — instead of one source_tier?

The case for. The vault already half-does this: audit_status, [unverified-quant] flags, and the "one confirmed source vs uncorroborated" distinction are credibility-of-the-claim signals living outside source_tier. Making the second axis explicit could sharpen retrieval gating.

The case against. Trained analysts empirically cannot keep the two axes independent — they over-weight source track record and avoid inconsistent pairings. If humans (and, per AuthorityBench, models) can't hold the axes apart, a second field may add false precision rather than honesty. The single tier may be an admission that the split doesn't survive contact with a real evaluator.

What I'd need to answer it. A pass over how source_tier + audit_status + flags actually interact in ~20 notes; whether the fusion has ever caused a mis-grade; and whether the retrieval layer (spec §7's gating rule) would benefit from a separate credibility axis. This is a design decision for Cali, not a unilateral schema change.

Candidate next move. Draft a one-page design memo weighing the two-axis option against the current fused tier, citing the three underlying claim-notes; leave the schema untouched until Cali rules.

Progress

2026-07-21 (partial — stays open). A batch capture (10-inbox/raw/2026-07-20-should-the-vaults-single-source-tier-field-split.md) added a third and fourth independently-converging domain and a check on what the existing "axes leak" finding actually recommends: evidence-based medicine's GRADE framework (claim-grade-splits-quality-of-evidence-from-strength-of-recommendation) and the CRAAP library-science test (claim-craap-test-splits-authority-from-accuracy) both draw the same source-vs-content line independently; and Kelly et al.'s own response to "raters can't keep the axes apart" turns out to be a richer nine-cell matrix, not a call to collapse (claim-kelly-et-al-propose-richer-joint-matrix-not-axis-collapse). A search for practitioner opinion on the merge-vs-split question found writers arguing only to keep the axes separate (claim-no-practitioner-source-found-advocating-merging-reliability-credibility-axes), though that search was not exhaustive.

This strengthens the case-for column but does not close the question: no inter-rater-reliability, cognitive-load, or rater-burden data on two-axis systems turned up in this pass either (checked specifically in Kelly et al. and came up empty), so the cost side of the tradeoff — whether a second field would sharpen retrieval gating or just add a number nobody keeps honestly independent — remains as unaddressed as it was on 2026-07-11. Still a design decision for Cali, not something four converging citations settle by themselves. Left open.

2026-08-07 (partial — stays open, new third option surfaced). Reading Benjamin Icard's own 2024 primary (claim-icard-2024-dynamic-logic-makes-credibility-primary-reliability-secondary) shows the "Icard (2023, 2024)" citation Kelly et al. point to is not the independence-preserving richer-matrix design the vault's existing note characterized it as (that characterization is accurate to what Kelly et al. say, just one citation-hop short of what Icard actually built). Icard's own proposal is a third shape neither column above considered: two named quantities, formally not independent by construction, with reliability demoted to a dynamic update operator on credibility rather than a coequal second axis. This doesn't answer the design question — whether the vault's source_tier should fuse, split, or asymmetrically-update is still Cali's call — but it means "split into two independent axes" was never the only alternative to fusion, and any design memo on this question should now weigh Icard's asymmetric-update shape alongside fuse/split. Left open.

2026-08-07 (partial — stays open, the leak now has a century-old name). A hop capture landing Thorndike's 1920 primary (claim-thorndike-1920-halo-effect-ratings-too-high-and-too-even, observation-halo-effect-names-the-2025-reliability-credibility-leak) identifies the "raters can't keep the axes apart" evidence in the case-against column as an instance of the halo effect — a robust, cross-domain, century-old psychometric bias, not a quirk of intelligence analysts. This strengthens the case-against a two-independent-axes split: the axes leak not because a particular evaluator population is careless but because a global impression contaminating independent trait ratings is a general and durable feature of human judgment. It does not settle the design question (fuse vs. split vs. Icard's asymmetric update remains Cali's call), and it says nothing about the still-unaddressed cost side (rater burden, whether a second field sharpens retrieval gating). Any design memo should now note that the leak it must design around is the halo effect by name. Left open.

2026-08-15 (partial — stays open, a third design dimension surfaced). Reading Samet (1975) at the primary (promotion of the 2026-08-11 channel-capacity hop) adds a dimension neither the fuse/split nor the Icard asymmetric-update framing considered: granularity per axis. The Admiralty Code's real in-house critique was not "too many axes" but "too few rungs" — Samet argued the scales were too coarse and should be finer, not fused (claim-samet-1975-argued-for-more-rating-categories-not-fewer), grounding the argument in Bendig (1954) and the information-theoretic ceiling Miller (1956) named as channel capacity (lineage: observation-admiralty-code-scale-length-descends-from-millers-channel-capacity). Any design memo on this question should now weigh scale length — how many levels each field carries — alongside fuse-vs-split-vs-asymmetric-update. This does not settle the design question (still Cali's call) and says nothing about the still-unaddressed cost side. Left open.

2026-08-18 (partial — stays open, correcting the entry immediately above and adding one more data point on the "how many axes" question). The 2026-08-15 entry's summary of Samet — "argued the scales were too coarse and should be finer, not fused" — is stale relative to a correction already applied to claim-samet-1975-argued-for-more-rating-categories-not-fewer on 2026-08-16: Samet's own implications section (p. 20) explicitly recommends fusion as well as finer resolution — "the two-dimensional evaluation should be replaced" by a single quantitative rating, with his subjects voting 21–16 in favor. So Samet is not a clean example of "finer, not fused"; he is a Tier-1 practitioner source arguing for both at once — one axis, made finer. A same-day bridge-check (observation-kelly-samet-cosine-pairing-real-link-opposite-axis-prescription) adds that Kelly et al. (2025) cite Samet directly in their own §2 for the axes-leak evidence, yet their discussion's own forward pointer (Icard's richer joint matrix) runs the opposite structural direction — more explicit joint categories across two axes, not one fused axis. So the "add structure" and "fuse the axes" columns above both now have their strongest respective advocates reading the same underlying evidence and prescribing oppositely, inside literatures that do not engage each other's specific fix. This narrows what a design memo can claim ("the field agrees more granularity helps" is not supportable — granularity-of-what remains contested) without resolving fuse vs. split vs. asymmetric-update, which is still Cali's call. Left open.

Correction appended 2026-09-13 (propagation-repair) Propagating the CORRECTED 2026-07-22 cross-model audit (auditor claude-fable-5, writer claude-sonnet-5) of claim-kelly-et-al-propose-richer-joint-matrix-not-axis-collapse. The progress entries above stand character-for-character — this file records the answer as it was given — but the 2026-07-21 entry's characterization of the nine-cell matrix is superseded.

Was: "Kelly et al.'s own response to 'raters can't keep the axes apart' turns out to be a richer nine-cell matrix, not a call to collapse" — the matrix framed as Kelly et al.'s own prescription, answering their own finding.

Now: two separate attributions in that sentence were too strong. (1) The 3 × 3 matrix is Icard's proposal (Icard 2023, 2024), which Kelly et al. merely point to in their general discussion as a comparison target for future work — "future studies could compare the reliability and perceived usefulness of the Admiralty Code to alternative methods that encode qualitative meaning at the 'cell' level" — not their own prescription. The claim note's verb was softened from "propose" to "point to" accordingly. (2) The "raters can't keep the axes apart" evidence is prior work Kelly et al. review in their §2 (Baker et al. 1968; Miron et al. 1978; Samet 1975; Mandel et al. 2023), not their own experimental finding; their experiments concern decoding, adding that trustworthiness ratings over-weight source reliability. So it is not "their own response" to "their own" finding on either side. The same audit also fixed the quotation, which was not verbatim despite the capture's claim, against the publisher HTML and the PDF (doi:10.1017/jdm.2025.10007).

What did not move. The directional point the 2026-07-21 entry drew on is unaffected: the paper's discussion, faced with the axes-leak evidence, still runs toward more explicit joint granularity rather than a single collapsed score. So that entry's conclusion — strengthens the case-for column, does not close the question — stands, as does its note that no inter-rater-reliability or rater-burden data turned up. The claim note's filename is unchanged, so the link above still resolves. The 2026-08-07 and 2026-08-18 entries already use the corrected voice ("the 'Icard (2023, 2024)' citation Kelly et al. point to", "their discussion's own forward pointer") and need no annotation; the 2026-08-07 entry's separate finding — that Icard's own primary builds an asymmetric-update shape rather than two independent axes — is a different, later correction and is untouched here.

No routing change. status: open preserved: this correction adjusts whose prescription the nine cells are, not whether the vault's source_tier should fuse, split, or asymmetrically update. Still Cali's call, and the cost side remains unaddressed.

· raw markdown

Should the vault's single source_tier field split into two axes — source reliability separate from claim credibility — the way intelligence doctrine and RAG both do?

Progress

Linked from