US intelligence doctrine tells analysts to rate probability and confidence independently, but Kelly et al. (2025) call that guidance "ill advised" because probability bounds confidence
The U.S. Office of the Director of National Intelligence (ODNI) instructs analysts to assign a probability rating (how likely an event is) and a confidence rating (how much the analyst trusts that estimate) as separate, independent quantities. Kelly, Budescu, Dhami & Mandel (2025), in the same Judgment and Decision Making paper the vault already cites for the finding that raters cannot keep source reliability and information credibility apart, argue in passing that this particular independence is not merely psychologically hard to honor but structurally constrained by the numbers themselves: "probabilities do, in fact, put boundaries on confidence levels." Their worked case is a 99% estimate — "an event that is judged to have a 99% chance of occurring can have at most an upper bound of 100% on a confidence interval around the estimate. Such a small room for error would be inconsistent with expressing 'low confidence'." From that they conclude: "the guidance from the intelligence community that such a probability could be coupled with low confidence is ill advised, and recent research shows that analysts and nonexperts alike do treat probability and confidence as related constructs."
The "recent research" cited is Irwin & Mandel (2023), "Communicating Uncertainty in National Security Intelligence" (Risk Analysis, 43, 943–957) — not, as this note said until 2026-08-11, "Probability or confidence, a distinction without a difference?" (a different, unrelated paper solely authored by Robert Levine; see claim-kelly-irwin-mandel-citation-misattributed-to-levine-2023). Read directly, Irwin & Mandel's paper confirms the empirical leg in both directions — stated confidence shifted analysts' inferred probability, and probability location shifted inferred confidence — resolving question-verify-irwin-mandel-2023-probability-confidence-cue.
The finding is a third instance of a pattern the vault has been accumulating: an evaluative schema prescribes two axes it declares independent, and independence turns out to be the assumption that fails. The other two are reliability/credibility (the Admiralty Code's two axes, shown to leak in the companion Kelly finding) and source-weight/relevance in machine retrieval (RA-RAG, sitting near Condorcet's independence precondition and its empirical failure in LLM ensembling). Here the colliding axes are probability and confidence, and — unlike the other two, which fail in human judgment — the objection is lodged against the numbers rather than against the raters: the doctrine asks for a pairing the probability value already constrains. It joins the same cluster as Kelly et al.'s discussion of richer joint categories and the broader two-axis convergence across intelligence doctrine and retrieval research. Co-author David R. Mandel runs the DRDC research program on analysts' forecasting accuracy that both the probability/confidence papers grow out of; Megan O. Kelly is the lead author.