---
id: "20260807-0955-hop-probability-confidence-not-independent"
title: "US intelligence doctrine tells analysts to judge probability and confidence independently — Kelly et al. (2025) call the guidance itself incoherent, extending the vault's independence-collapse pattern to a third axis pair"
type: "capture"
status: "promoted"
promoted_to: ["30-notes/claim-kelly-2025-odni-probability-confidence-guidance-incoherent.md"]
promoted_questions: ["50-questions/question-verify-irwin-mandel-2023-probability-confidence-cue.md"]
promoted_entities: ["40-entities/entity-megan-o-kelly.md (new hub)","40-entities/entity-david-r-mandel.md (updated — dated line appended)"]
not_promoted: ["Irwin & Mandel (2023) specific mechanism (confidence shifts inferred probability more than interval width) — read only via secondary/WebSearch description this session, flagged [unverified-mechanism]; folded into the ODNI note as a lead and routed to question-verify-irwin-mandel-2023-probability-confidence-cue rather than written as a sourced claim. Quote-provenance rule bars a claim on a summary-layer rendering.","Kelly et al. Experiment finding (1): intraindividual reliability best for MEDIUM consistency, worst for maximally consistent reliability/credibility pairs, contradicting the attribute-consistency hypothesis — a real Tier-1 result, but out of this capture's probability/confidence scope, drawn from the same paper that already anchors claim-source-reliability-and-credibility-are-not-judged-independently, and the capture supplies no verbatim quote to ground it. Left for a future dedicated note if it recurs.","Kelly et al. finding (2): trustworthiness ratings depend more on source reliability than information credibility — already covered by claim-source-reliability-and-credibility-are-not-judged-independently ('over-weight the source's track record'). Duplicate; not re-promoted.","The 'third instance' meta-pattern (prescribed independence collapses across three axis-pairs) — folded into the ODNI note's body and commentary with dense wikilinks rather than spun into a separate observation note; the vault already carries nearby observation notes and a flagged missing MOC for this cluster. Flagged in 00-meta/seek-flags.md as a possible dedicated observation/MOC once the shape settles.","Entity: Daniel Irwin (co-author) — unknown to vault, rests on an unread/unverified paper; not promoted (entity-page-spec: when unsure, don't).","Entity: Office of the Director of National Intelligence — real established org but appears in only one note and is not yet recurring/load-bearing; held as a mention, not promoted."]
origin: "hop-batch"
writer_model: "claude-sonnet-5"
date_created: "2026-08-07T00:00:00.000Z"
hop_chain: ["seed: claim-reliability-aware-rag-estimates-source-reliability-separately-from-relevance <-> claim-source-reliability-and-credibility-are-not-judged-independently (cosine 0.89, bipartite ml-ai/epistemology pair)","seed pair -> Kelly et al. (2025) 'The effect of source reliability and information credibility on judgments of information quality in intelligence analysis', Judgment and Decision Making (mechanism hook: chased the Tier-1 primary behind the vault's own credibility-independence claim; max_cosine n/a, direct source read)","Kelly et al. 2025 -> vault_bridge check on the RA-RAG/Condorcet cluster (mechanism hook: confirmed the vault had already built the RA-RAG <-> Kelly independence bridge on 2026-08-02 via claim-condorcet-1785-jury-theorem-requires-independent-voters and claim-lefort-2024-llm-ensembling-marginal-gains-non-independent-errors; max_cosine 0.811, bridge_candidate=false — the direct seed-pair link is still missing but two independent hub notes now carry it)","Kelly et al. 2025's ODNI critique -> Irwin & Mandel (2023) 'Probability or confidence, a distinction without a difference?', Intelligence and National Security (surprising-claim hook, followed via Kelly et al.'s own citation and WebSearch abstract; max_cosine 0.691, frontier P11.1)","Irwin & Mandel 2023 -> David R. Mandel's DRDC forecasting-accuracy research program (person-behind-the-thing hook, zoom-out via web bio; David R. Mandel already known to vault, mention_count=1, no entity page)"]
novelty_max_cosine: 0.691
tags: ["intelligence-tradecraft","epistemics","source-evaluation","forecasting","independence","cognitive-bias"]
source_url: "https://www.cambridge.org/core/services/aop-cambridge-core/content/view/E67548E8010A47345C3439D45D9EC6B3/S1930297525100077a.pdf/the-effect-of-source-reliability-and-information-credibility-on-judgments-of-information-quality-in-intelligence-analysis.pdf"
source_title: "The effect of source reliability and information credibility on judgments of information quality in intelligence analysis"
source_author: "Megan O. Kelly, David V. Budescu, Mandeep Dhami, David R. Mandel"
source_date: 2025
source_venue: "Judgment and Decision Making, 20:e36"
source_tier: 1
source_sha: "3dbc02831d749dcbf4029b70e9a3064f64a66dc0c8dac91c1fbfc7a371cf836c"
source_quote: "the guidance from the intelligence community that such a probability could be coupled with low confidence is ill advised, and recent research shows that analysts and nonexperts alike do treat probability and confidence as related constructs"
seek_code_commit: "649b1a4"
---


Kelly et al. (2025), in the same paper the vault already cites for the finding that evaluators can't keep source reliability and information credibility apart, make a second, sharper claim in passing: the U.S. Office of the Director of National Intelligence instructs analysts to assign **probability** and **confidence** ratings independently of one another — but the two are not actually independent quantities. A 99%-probable event has, by construction, at most a 1-point upper bound on its error margin, which is "inconsistent with expressing 'low confidence.'" Kelly et al. write: "the guidance from the intelligence community that such a probability could be coupled with low confidence is ill advised, and recent research shows that analysts and nonexperts alike do treat probability and confidence as related constructs" (source_tier 1, direct PDF read).

The cited "recent research" is Irwin & Mandel (2023), "Probability or confidence, a distinction without a difference?" (*Intelligence and National Security*) — not read directly this session; recorded here as a lead, `[unverified-mechanism -- needs primary]` for its own specific numbers. David R. Mandel, co-author of both papers, is a DRDC Toronto senior scientist whose research program tracks the forecasting accuracy of intelligence analysts over years-long horizons.

This is a **third instance** of the vault's recurring pattern — human evaluators cannot keep two "independent" meta-informational axes apart — joining reliability/credibility (Kelly et al., Admiralty Code) and source-weight/relevance (RA-RAG, Condorcet Jury Theorem). Here the colliding axes are probability and confidence, and the incoherence is not just psychological but mathematical: the guidance asks for something the numbers themselves forbid.

## Why this was hop-worthy

The seed pair's actual bridge (Condorcet's independence precondition) turned out to already exist in the vault, built five days ago from an unrelated chain — the more interesting find was a third, un-vaulted instance of the same collapse, one level over in the same paper.

## Further leads
- Irwin & Mandel (2023), "Probability or confidence, a distinction without a difference?" — not yet read primary; would upgrade this note's central claim past Tier-1-by-citation.
- SSRN working paper "How Intelligence Organizations Communicate Confidence (Unclearly)" (Irwin & Mandel) may be an open-access preprint of the same work.

## Entity candidates
- David R. Mandel — person — DRDC Toronto senior scientist, co-author of both the Kelly et al. and Irwin & Mandel papers; already 1 vault mention, no entity page; his forecasting-accuracy research program is untouched territory for the vault.
- Daniel Irwin — person — co-author of the probability/confidence "distinction without a difference" critique; unknown to vault.
- Megan O. Kelly — person — lead author of the Tier-1 paper this capture and two existing vault claims rest on; already load-bearing for 2+ notes, no entity page yet.
- Office of the Director of National Intelligence — concept/org — the doctrine-setting body whose independence guidance is the target of the critique; unknown to vault.

## Hop chain

Hop 1: Kelly, Budescu, Dhami & Mandel (2025), "The effect of source reliability and information credibility on judgments of information quality in intelligence analysis" — https://www.cambridge.org/core/journals/judgment-and-decision-making/article/effect-of-source-reliability-and-information-credibility-on-judgments-of-information-quality-in-intelligence-analysis/E67548E8010A47345C3439D45D9EC6B3
- Hook type: mechanism question
- Hook: the vault's existing claim-note quotes one sentence of this paper's abstract; the full text has the actual experimental mechanism and a second, unquoted critique of ODNI doctrine.
- Why followed: chase the Tier-1 primary behind an existing vault claim before hopping onward, per NAME/mechanism check.
- Key findings: (1) intraindividual reliability was worst for *maximally consistent* reliability/credibility pairs and best for *medium* consistency — the opposite of the "attribute consistency hypothesis" the paper set out to test; (2) trustworthiness ratings depend more on source reliability than information credibility; (3) the paper explicitly calls ODNI's probability/confidence independence guidance "ill advised."
- Surprise: expected highly consistent (matching) reliability/credibility ratings to produce the most stable judgments — found medium consistency was significantly more reliable than either low or high consistency across both experiments.

Hop 2: vault_bridge check on the RA-RAG / Condorcet cluster (internal, no external URL)
- Hook type: cross-domain bridge (checking one already flagged in the vault)
- Hook: RA-RAG's "weighted majority voting" and the Kelly et al. independence failure both sit near Condorcet's 1785 Jury Theorem and Lefort et al.'s 2024 LLM-ensembling paper in the vault's own embedding space.
- Why followed: the seed's own framing asked whether the RA-RAG/Kelly resemblance was a real mechanism; the Condorcet cluster is exactly that mechanism, already captured.
- Key findings: claim-lefort-2024-llm-ensembling-marginal-gains-non-independent-errors already wikilinks directly to claim-source-reliability-and-credibility-are-not-judged-independently, and both link to claim-condorcet-1785-jury-theorem-requires-independent-voters, which in turn names RA-RAG explicitly. The seed pair is connected — through two intermediary hub notes built on 2026-07-11 and 2026-08-02 — but still has no direct wikilink to each other five days later.
- Surprise: expected to be finding a genuinely new connection for the seed pair — found the vault had already built the connecting mechanism, unprompted, in an unrelated chain five days before this session.

Hop 3: Irwin & Mandel (2023), "Probability or confidence, a distinction without a difference?" — https://www.tandfonline.com/doi/full/10.1080/02684527.2023.2276582 (abstract/description via WebSearch; not fetched primary this session)
- Hook type: surprising claim
- Hook: the title itself is a reversal — a doctrine built on distinguishing two things, accused by its own research community of drawing a distinction that doesn't exist.
- Why followed: Kelly et al. cite this paper as the direct evidence for the ODNI critique; the title alone signals a load-bearing, quotable finding.
- Key findings (via secondary description, not primary read): experiments with intelligence analysts found that varying stated confidence level shifted analysts' inferred event probabilities more than it shifted the width of their confidence intervals — i.e., confidence functions as a probability cue rather than a genuinely separate quantity.

Hop 4: David R. Mandel — DRDC Toronto bio (https://mandel.socialpsychology.org/, via WebSearch)
- Hook type: the person behind the thing
- Hook: Mandel is the common author across both papers in this chain and the vault's pre-existing Kelly-et-al claim; DRDC's "Anticipatory Intelligence Pillar" work on forecasting accuracy is untouched by the vault.
- Why followed: zoom-out after three hops of increasingly specific claims; alternation per protocol.
- Key findings: Mandel is a DRDC senior scientist and cross-appointed professor (Waterloo, York) running long-term studies tracking intelligence analysts' forecast accuracy; chaired a NATO panel on communicating uncertainty in intelligence. No cross-domain landing back to AI found this session — noted as a future lead rather than forced.

Saved hooks not followed:
- Cognitive Reflection Task (Frederick, 2005), used in Kelly et al. Experiment 2 — from Kelly et al. 2025 — already well-known outside the vault's frontier, not pursued.
- "Independent, Well-Trained, Uniformly Biased" (IWTUB), Lefort et al.'s coinage for the Condorcet-ideal voter set — from claim-lefort-2024 — interesting term but already captured in an existing vault note, not a first encounter.
- SSRN "How Intelligence Organizations Communicate Confidence (Unclearly)" — surfaced alongside Irwin & Mandel 2023 — likely an open preprint of the same work, saved as a future primary-source fetch.

post-worthy: maybe — the probability/confidence claim is fresh and frontier-scored, but its strongest evidence (Irwin & Mandel 2023) is still a secondary citation this session; worth revisiting once that primary is read directly.