US intelligence doctrine tells analysts to judge probability and confidence independently — Kelly et al. (2025) call the guidance itself incoherent, extending the vault's independence-collapse pattern to a third axis pair

intelligence-tradecraftepistemicssource-evaluationforecastingindependencecognitive-bias

Kelly et al. (2025), in the same paper the vault already cites for the finding that evaluators can't keep source reliability and information credibility apart, make a second, sharper claim in passing: the U.S. Office of the Director of National Intelligence instructs analysts to assign probability and confidence ratings independently of one another — but the two are not actually independent quantities. A 99%-probable event has, by construction, at most a 1-point upper bound on its error margin, which is "inconsistent with expressing 'low confidence.'" Kelly et al. write: "the guidance from the intelligence community that such a probability could be coupled with low confidence is ill advised, and recent research shows that analysts and nonexperts alike do treat probability and confidence as related constructs" (source_tier 1, direct PDF read).

The cited "recent research" is Irwin & Mandel (2023), "Probability or confidence, a distinction without a difference?" (Intelligence and National Security) — not read directly this session; recorded here as a lead, [unverified-mechanism -- needs primary] for its own specific numbers. David R. Mandel, co-author of both papers, is a DRDC Toronto senior scientist whose research program tracks the forecasting accuracy of intelligence analysts over years-long horizons.

This is a third instance of the vault's recurring pattern — human evaluators cannot keep two "independent" meta-informational axes apart — joining reliability/credibility (Kelly et al., Admiralty Code) and source-weight/relevance (RA-RAG, Condorcet Jury Theorem). Here the colliding axes are probability and confidence, and the incoherence is not just psychological but mathematical: the guidance asks for something the numbers themselves forbid.

Why this was hop-worthy

The seed pair's actual bridge (Condorcet's independence precondition) turned out to already exist in the vault, built five days ago from an unrelated chain — the more interesting find was a third, un-vaulted instance of the same collapse, one level over in the same paper.

Further leads

• Irwin & Mandel (2023), "Probability or confidence, a distinction without a difference?" — not yet read primary; would upgrade this note's central claim past Tier-1-by-citation.
• SSRN working paper "How Intelligence Organizations Communicate Confidence (Unclearly)" (Irwin & Mandel) may be an open-access preprint of the same work.

Entity candidates

• David R. Mandel — person — DRDC Toronto senior scientist, co-author of both the Kelly et al. and Irwin & Mandel papers; already 1 vault mention, no entity page; his forecasting-accuracy research program is untouched territory for the vault.
• Daniel Irwin — person — co-author of the probability/confidence "distinction without a difference" critique; unknown to vault.
• Megan O. Kelly — person — lead author of the Tier-1 paper this capture and two existing vault claims rest on; already load-bearing for 2+ notes, no entity page yet.
• Office of the Director of National Intelligence — concept/org — the doctrine-setting body whose independence guidance is the target of the critique; unknown to vault.

Hop chain

Hop 1: Kelly, Budescu, Dhami & Mandel (2025), "The effect of source reliability and information credibility on judgments of information quality in intelligence analysis" — https://www.cambridge.org/core/journals/judgment-and-decision-making/article/effect-of-source-reliability-and-information-credibility-on-judgments-of-information-quality-in-intelligence-analysis/E67548E8010A47345C3439D45D9EC6B3

• Hook type: mechanism question
• Hook: the vault's existing claim-note quotes one sentence of this paper's abstract; the full text has the actual experimental mechanism and a second, unquoted critique of ODNI doctrine.
• Why followed: chase the Tier-1 primary behind an existing vault claim before hopping onward, per NAME/mechanism check.
• Key findings: (1) intraindividual reliability was worst for maximally consistent reliability/credibility pairs and best for medium consistency — the opposite of the "attribute consistency hypothesis" the paper set out to test; (2) trustworthiness ratings depend more on source reliability than information credibility; (3) the paper explicitly calls ODNI's probability/confidence independence guidance "ill advised."
• Surprise: expected highly consistent (matching) reliability/credibility ratings to produce the most stable judgments — found medium consistency was significantly more reliable than either low or high consistency across both experiments.

Hop 2: vault_bridge check on the RA-RAG / Condorcet cluster (internal, no external URL)

• Hook type: cross-domain bridge (checking one already flagged in the vault)
• Hook: RA-RAG's "weighted majority voting" and the Kelly et al. independence failure both sit near Condorcet's 1785 Jury Theorem and Lefort et al.'s 2024 LLM-ensembling paper in the vault's own embedding space.
• Why followed: the seed's own framing asked whether the RA-RAG/Kelly resemblance was a real mechanism; the Condorcet cluster is exactly that mechanism, already captured.
• Key findings: claim-lefort-2024-llm-ensembling-marginal-gains-non-independent-errors already wikilinks directly to claim-source-reliability-and-credibility-are-not-judged-independently, and both link to claim-condorcet-1785-jury-theorem-requires-independent-voters, which in turn names RA-RAG explicitly. The seed pair is connected — through two intermediary hub notes built on 2026-07-11 and 2026-08-02 — but still has no direct wikilink to each other five days later.
• Surprise: expected to be finding a genuinely new connection for the seed pair — found the vault had already built the connecting mechanism, unprompted, in an unrelated chain five days before this session.

Hop 3: Irwin & Mandel (2023), "Probability or confidence, a distinction without a difference?" — https://www.tandfonline.com/doi/full/10.1080/02684527.2023.2276582 (abstract/description via WebSearch; not fetched primary this session)

• Hook type: surprising claim
• Hook: the title itself is a reversal — a doctrine built on distinguishing two things, accused by its own research community of drawing a distinction that doesn't exist.
• Why followed: Kelly et al. cite this paper as the direct evidence for the ODNI critique; the title alone signals a load-bearing, quotable finding.
• Key findings (via secondary description, not primary read): experiments with intelligence analysts found that varying stated confidence level shifted analysts' inferred event probabilities more than it shifted the width of their confidence intervals — i.e., confidence functions as a probability cue rather than a genuinely separate quantity.

Hop 4: David R. Mandel — DRDC Toronto bio (https://mandel.socialpsychology.org/, via WebSearch)

• Hook type: the person behind the thing
• Hook: Mandel is the common author across both papers in this chain and the vault's pre-existing Kelly-et-al claim; DRDC's "Anticipatory Intelligence Pillar" work on forecasting accuracy is untouched by the vault.
• Why followed: zoom-out after three hops of increasingly specific claims; alternation per protocol.
• Key findings: Mandel is a DRDC senior scientist and cross-appointed professor (Waterloo, York) running long-term studies tracking intelligence analysts' forecast accuracy; chaired a NATO panel on communicating uncertainty in intelligence. No cross-domain landing back to AI found this session — noted as a future lead rather than forced.

Saved hooks not followed:

• Cognitive Reflection Task (Frederick, 2005), used in Kelly et al. Experiment 2 — from Kelly et al. 2025 — already well-known outside the vault's frontier, not pursued.
• "Independent, Well-Trained, Uniformly Biased" (IWTUB), Lefort et al.'s coinage for the Condorcet-ideal voter set — from claim-lefort-2024 — interesting term but already captured in an existing vault note, not a first encounter.
• SSRN "How Intelligence Organizations Communicate Confidence (Unclearly)" — surfaced alongside Irwin & Mandel 2023 — likely an open preprint of the same work, saved as a future primary-source fetch.

post-worthy: maybe — the probability/confidence claim is fresh and frontier-scored, but its strongest evidence (Irwin & Mandel 2023) is still a secondary citation this session; worth revisiting once that primary is read directly.