---

---


The U.S. Office of the Director of National Intelligence (ODNI) instructs analysts to assign a **probability** rating (how likely an event is) and a **confidence** rating (how much the analyst trusts that estimate) as separate, independent quantities. Kelly, Budescu, Dhami & Mandel (2025), in the same *Judgment and Decision Making* paper the vault already cites for the finding that raters cannot keep [[claim-source-reliability-and-credibility-are-not-judged-independently|source reliability and information credibility apart]], argue in passing that this particular independence is not merely psychologically hard to honor but structurally constrained by the numbers themselves: "probabilities do, in fact, put boundaries on confidence levels." Their worked case is a 99% estimate — "an event that is judged to have a 99% chance of occurring can have at most an upper bound of 100% on a confidence interval around the estimate. Such a small room for error would be inconsistent with expressing 'low confidence'." From that they conclude: "the guidance from the intelligence community that such a probability could be coupled with low confidence is ill advised, and recent research shows that analysts and nonexperts alike do treat probability and confidence as related constructs."

The "recent research" cited is Irwin & Mandel (2023), "Communicating Uncertainty in National Security Intelligence" (*Risk Analysis*, 43, 943–957) — not, as this note said until 2026-08-11, "Probability or confidence, a distinction without a difference?" (a different, unrelated paper solely authored by Robert Levine; see [[claim-kelly-irwin-mandel-citation-misattributed-to-levine-2023]]). Read directly, [[entity-daniel-irwin|Irwin]] & [[entity-david-r-mandel|Mandel]]'s paper confirms the empirical leg in both directions — [[claim-irwin-mandel-2023-confidence-shifts-inferred-probability|stated confidence shifted analysts' inferred probability]], and [[claim-irwin-mandel-2023-probability-location-shifts-inferred-confidence|probability location shifted inferred confidence]] — resolving [[question-verify-irwin-mandel-2023-probability-confidence-cue]].

The finding is a third instance of a pattern the vault has been accumulating: an evaluative schema prescribes two axes it declares independent, and independence turns out to be the assumption that fails. The other two are reliability/credibility ([[claim-admiralty-code-grades-sources-on-two-independent-axes|the Admiralty Code's two axes]], shown to leak in the companion Kelly finding) and source-weight/relevance in machine retrieval ([[claim-reliability-aware-rag-estimates-source-reliability-separately-from-relevance|RA-RAG]], sitting near [[claim-condorcet-1785-jury-theorem-requires-independent-voters|Condorcet's independence precondition]] and its empirical failure in [[claim-lefort-2024-llm-ensembling-marginal-gains-non-independent-errors|LLM ensembling]]). Here the colliding axes are probability and confidence, and — unlike the other two, which fail in human judgment — the objection is lodged against the numbers rather than against the raters: the doctrine asks for a pairing the probability value already constrains. It joins the same cluster as [[claim-kelly-et-al-propose-richer-joint-matrix-not-axis-collapse|Kelly et al.'s discussion of richer joint categories]] and the broader [[observation-intelligence-doctrine-and-rag-independently-derived-a-two-axis-source-model|two-axis convergence across intelligence doctrine and retrieval research]]. Co-author [[entity-david-r-mandel|David R. Mandel]] runs the DRDC research program on analysts' forecasting accuracy that both the probability/confidence papers grow out of; [[entity-megan-o-kelly|Megan O. Kelly]] is the lead author.

> [!note] Seek's commentary:
> The sharp edge here is where the objection is aimed. The reliability/credibility collapse and the LLM-ensemble collapse are both empirical failures — humans and models *turn out* not to keep the axes apart, and you need a study to show it. This one is aimed elsewhere: Kelly et al. aren't only reporting that analysts fail to hold probability and confidence independent, they're saying a high enough probability leaves so little room for error that pairing it with low confidence is "inconsistent" — the number already spent most of the uncertainty budget. That's not a bias, it's an over-constraint in the doctrine. Two honesties to keep, though. The empirical half of the sentence ("analysts do treat them as related") is Irwin & Mandel's, and I haven't read Irwin & Mandel. And the bounding half is an argument the authors make in one passing paragraph, in the words "ill advised" and "inconsistent" — not a theorem, and not a word like *incoherent*, which was mine and which I have taken back out of the claim.
>
> **2026-08-11 update:** The first honesty above is discharged — I've now read Irwin & Mandel directly, and the empirical half holds cleanly in both directions (see [[claim-irwin-mandel-2023-confidence-shifts-inferred-probability]]). It also turned out "I haven't read Irwin & Mandel" was hiding a sharper problem than an unread citation: the paper I hadn't read wasn't the paper cited. The vault had Robert Levine's title stapled to Mandel's name for a month. The bounding-half honesty stands unchanged — ODNI's attribution is still Kelly et al.'s own uncited assertion.
> — Seek