Does Irwin & Mandel (2023) actually show that intelligence analysts treat confidence as a probability cue rather than a separate quantity?

intelligence-tradecraftepistemicsforecastingindependenceprobability-confidencesource-evaluationcitation-correctiondavid-r-mandelsourcing-floor

Short answer: yes, verbatim, on both directions of the effect — intelligence analysts (and non-experts) in Irwin & Mandel's actual 2023 experiments do not hold confidence and probability apart; raising stated confidence pulls inferred probability estimates upward, and probability level itself gets read back as a confidence signal. That resolves question-verify-irwin-mandel-2023-probability-confidence-cue as far as a direct primary read allows. But getting to that answer required first fixing which paper the question was even about: the vault's citation chain had drifted onto a different, unrelated 2023 paper with a similar title — a correction that matters more than any single number below, since claim-kelly-2025-odni-probability-confidence-guidance-incoherent's empirical leg was resting on it.

Claim: The "Irwin & Mandel (2023)" paper actually cited by Kelly et al. (2025) is the Risk Analysis empirical study, not "Probability or confidence, a distinction without a difference?" — a citation-chain error the vault has carried since 2026-08-07

Claim type: historical/bibliographic (which document a citation points to). Tier 3–4 acceptable per the floor; met at Tier 1 (Crossref registry metadata + a direct read of both candidate papers' own bibliographic self-identification).

Three existing vault files (entity-david-r-mandel, claim-kelly-2025-odni-probability-confidence-guidance-incoherent, and question-verify-irwin-mandel-2023-probability-confidence-cue) name the paper behind Kelly et al.'s "(Irwin and Mandel, 2023)" citation as "Probability or confidence, a distinction without a difference?" (Intelligence and National Security). Crossref's own metadata for that DOI (10.1080/02684527.2023.2276582) lists exactly one author: "given":"Robert","family":"Levine" — not Irwin, not Mandel. Levine's own reference list (also in the Crossref record) cites "Irwin D. and D. R. Mandel. 'How Intelligence Organizations Communicate Confidence (Unclearly).' DRDC – Toronto Research Centre. December 2018" as one of his sources — Levine is citing Irwin & Mandel's earlier work, not co-authoring with them. Reading Kelly et al.'s own reference list directly resolves which paper they actually meant:

"Irwin, D., & Mandel, D. R. (2023). Communicating uncertainty in national security intelligence: Expert and non-expert interpretations of and preferences for verbal and numeric formats. Risk Analysis, 43, 943–957."

This is the paper this capture reads below. The likely origin of the error: the 2026-08-07 hop capture's Hop 3 followed "Irwin & Mandel (2023), 'Probability or confidence, a distinction without a difference?'" via a WebSearch abstract description rather than a primary read or a check of Kelly et al.'s own bibliography — the search engine's summary conflated two same-year, same-topic, similarly-titled papers into one citation. Notably, the description the 2026-08-07 capture recorded under that mistaken title ("varying stated confidence level shifted analysts' inferred event probabilities more than it shifted the width of their confidence intervals") is in fact an accurate paraphrase of the correct paper's finding (see next claim) — the mechanism was right, the title and putative second author were wrong.

provenance: Crossref API record for DOI 10.1080/02684527.2023.2276582 (Tier 1, publisher-deposited); Kelly et al. (2025) reference list, p. 20 of the PDF pagination (Tier 1, direct extract_pdf read, sha matches vault's existing record).

Claim: In an experiment with N=41 professional intelligence analysts, raising the stated verbal confidence level (low/moderate/high) attached to a probability assessment significantly increased analysts' inferred numeric probability — not just their margin of error, as normative treatment of confidence would require

Claim type: quantitative + technical-mechanism. Tier 1–2 required; met at Tier 1 (direct extract_pdf read of the primary preprint, quote-check-grounded).

Irwin & Mandel's expert sample (21 Canadian intelligence analysts recruited in person, 20 more via a remote Qualtrics link distributed by their managers) were given a fixed hypothetical intelligence assessment using the term "likely," then asked to provide 95%-certainty probability ranges for that assessment under three conditions: low, moderate, and high stated confidence. If confidence functioned as a genuinely separate quantity — expressing only the analyst's margin of error — the midpoint of the inferred range should stay constant across confidence levels while only its width changes. Instead:

"Had experts treated probability and confidence as independent constructs, we would expect to observe invariant midpoint interpretations, yet midpoint interpretations substantially increased with each increase in confidence level" — F(1, 39) = 90.62, p < .001, partial η² = .699 (all pairwise comparisons between confidence levels significant, Fisher's LSD, α = .05).

The same pattern held, with an even larger sample, in the non-expert replication (n = 440 in Experiment 1, n = 624 in Experiment 2), and in Experiment 2 confidence level had a stronger effect on the inferred probability midpoint than the stated probability term itself did (two-way ANOVA, significant main effects of both confidence level and probability level, F[2,621]=38.43 and F[1,621]=33.83 respectively, both p<.001, plus a significant interaction). The paper's general discussion states the conclusion directly:

"Critically, we found that intelligence consumers do not treat confidence and probability as independent constructs."

provenance: Irwin & Mandel (2023), pp. 15–16 and 26 of the in-press pagination (PsyArXiv preprint, Tier 1, quote-check-grounded).

Claim: The same expert analysts also ran the effect in reverse — inferring an analyst's confidence level from where a stated probability interval fell on the 0–100% scale, not from the interval's width

Claim type: quantitative + technical-mechanism. Tier 1–2 required; met at Tier 1.

In a separate task, the same 41 analysts were shown two numeric probability intervals with an identical 20-point spread (20%–40% and 60%–80%, equidistant from the scale's midpoint) and asked what confidence level (low/moderate/high) each communicated. A genuinely independent confidence judgment should treat the two intervals identically, since spread — not location — is what confidence is supposed to encode. Analysts did not:

"experts did not separate the location of the probabilities from the width of the intervals. Specifically, they judged the range below 50% as indicating mainly low or moderate confidence, whereas they judged the range above 50% as indicating mainly moderate or high confidence." — Wilcoxon signed-rank test, Z = −4.69, p < .001.

Non-experts showed the identical pattern (Z = −14.56, p < .001), as did the independent Experiment 2 sample of 624 (Z = −15.17, p < .001). Together with the previous claim, this establishes the conflation runs both directions: confidence functions as a de facto probability cue (shifts the inferred event likelihood), and probability level functions as a de facto confidence cue (shifts the inferred trustworthiness of the estimate) — which is the specific empirical content behind Kelly et al.'s summary phrase "analysts and nonexperts alike do treat probability and confidence as related constructs."

provenance: Irwin & Mandel (2023), pp. 16–17 of the in-press pagination (Tier 1, quote-check-grounded).

Claim: Irwin & Mandel frame their laboratory result as corroborating an independently documented, real-world pattern — Friedman & Zeckhauser's (2012) analysis of declassified intelligence products

Claim type: historical (what the paper itself asserts about prior evidence). Tier 3–4 acceptable per the floor; met at Tier 1, since this is Irwin & Mandel's own stated framing, directly quoted.

The paper explicitly situates its lab finding against an earlier, non-experimental evidence source rather than presenting the conflation as a novel discovery:

"findings cohere with evidence from declassified intelligence products that analysts conflate ordinal confidence ratings with event probabilities (Friedman & Zeckhauser, 2012)."

This is the earlier work the "analysts conflate confidence and probability" claim is measured against — Friedman & Zeckhauser (2012), "Assessing uncertainty in intelligence" (Intelligence and National Security, 27(6), 824–847), examined real declassified assessments rather than lab-elicited judgments. Irwin & Mandel also cite a partial counterpoint from the same research pair's later work, noting "previous research suggests that experts and non-experts can distinguish between probability and specific components of confidence (Friedman & Zeckhauser, 2018)" before presenting their own results as showing the disjunction fails in practice regardless.

provenance: Irwin & Mandel (2023), p. 27 of the in-press pagination (Tier 1, quote-check-grounded); Friedman & Zeckhauser (2012) and (2018) citations are Irwin & Mandel's own reference-list entries, not independently read this session.

Central question status

Does Irwin & Mandel (2023) show that intelligence analysts treat confidence as a probability cue rather than a separate quantity? Confirmed, directly, at Tier 1 — once the correct paper is identified. The paper Kelly et al. (2025) actually cite as "Irwin and Mandel, 2023" is "Communicating Uncertainty in National Security Intelligence" (Risk Analysis), not "Probability or confidence, a distinction without a difference?" (which is a different, unrelated, single-authored 2023 paper by Robert Levine). Read directly, the correct paper's own N=41 expert intelligence-analyst sample shows the effect runs in both directions — confidence shifts inferred probability, and probability location shifts inferred confidence — and the authors themselves frame this as corroborating, not contradicting, prior evidence from real declassified intelligence products (Friedman & Zeckhauser, 2012).

Further leads

• Levine, R. (2023), "Probability or confidence, a distinction without a difference?" (Intelligence and National Security, 39(4), 729–741, doi:10.1080/02684527.2023.2276582) — the paper the vault previously miscredited to Irwin & Mandel; a genuinely distinct, unread voice on the same doctrinal question, citing an earlier Irwin & Mandel working paper (SSRN 3441302 / DRDC 2018) among its sources. [unverified — needs direct read]
• Irwin & Mandel (2023) also found that roughly ¼ of the 41 expert analysts and over half of 440–624 non-experts gave logically incoherent numeric translations (lower bound > best estimate, etc.) for the plain probability terms "likely" and "unlikely" when elicitation was split by intervening tasks — a related but distinct instability finding about verbal-probability translation, not about the confidence/probability conflation specifically.
• Format-preference finding (same paper): expert analysts were split roughly evenly between preferring verbal and numeric confidence/probability formats, while non-experts leaned numeric; both samples rated the numeric format as more informative regardless of stated preference.
• Friedman, J. A., & Zeckhauser, R. (2012), "Assessing uncertainty in intelligence" (Intelligence and National Security, 27(6), 824–847) — the earlier declassified-documents study Irwin & Mandel say their finding "cohere[s] with"; not read at primary this session, and the actual foundational comparison point for the whole cluster (see Entity candidates).
• Friedman, J. A., & Zeckhauser, R. (2018), "Analytic confidence and political decision-making" (Political Psychology, 39(5), 1069–1087) — cited by Irwin & Mandel as evidence that experts can sometimes distinguish probability from specific components of confidence, a partial counterpoint noted but not resolved in Irwin & Mandel's own discussion.
• Duke, [initials], et al. (2024), "Probability and confidence: How to improve communication of uncertainty about uncertainty in intelligence analysis" (Journal of Behavioral Decision Making), surfaced in search as a 2024 follow-on citing this cluster — not read.
• Gregory Treverton (former Chair, US National Intelligence Council), quoted secondhand inside Irwin & Mandel's own future-directions section: "In intelligence, most treatments of confidence are little more than hand-waves" (2022, p. 47) — a colorful, quotable line, but sourced here only via Irwin & Mandel's citation of Treverton, not an independent read of Treverton's own text.
• This capture's claim about Levine's paper's authorship (via Crossref) should be used to correct the 2026-08-07-derived text in entity-david-r-mandel and claim-kelly-2025-odni-probability-confidence-guidance-incoherent, both of which currently misname the Levine paper as an Irwin & Mandel work.

Entity candidates

• Jeffrey A. Friedman — person — co-author (with Zeckhauser) of the 2012 declassified-intelligence-products study Irwin & Mandel's 2023 lab finding is explicitly measured against ("findings cohere with evidence from declassified intelligence products..."); the earlier, foundational empirical anchor for the whole "analysts conflate confidence and probability" claim, not just a citation in passing. Not yet an entity in the vault. Flagged first per the standing blind-spot note.
• Richard Zeckhauser — person — co-author with Friedman on both the 2012 and 2018 papers cited by Irwin & Mandel; same foundational role as Friedman above.
• Daniel Irwin — person — co-author (with Mandel) of the actual paper this capture reads; already flagged once before (2026-08-07 hop) as unknown to the vault and not promoted; still no entity page.
• Robert Levine — person — sole author, per Crossref, of "Probability or confidence, a distinction without a difference?" (Intelligence and National Security, 2023) — previously miscredited in the vault as a Mandel/Irwin paper; a real, distinct researcher on the same doctrinal question whose own text the vault has never read.
• Gregory Treverton — person — former Chair, US National Intelligence Council; quoted inside Irwin & Mandel (2023) calling intelligence-community treatments of confidence "little more than hand-waves" — a secondhand quote, not independently verified against his own text.

Related, already-hubbed: entity-david-r-mandel (co-author of the paper this capture reads; entity page needs its 2026-08-07 "Updates" line corrected per the citation-chain fix above) and entity-megan-o-kelly (lead author of the paper whose citation started the chain). Cluster context: claim-kelly-2025-odni-probability-confidence-guidance-incoherent, claim-source-reliability-and-credibility-are-not-judged-independently, claim-kelly-et-al-propose-richer-joint-matrix-not-axis-collapse, observation-intelligence-doctrine-and-rag-independently-derived-a-two-axis-source-model, moc-two-axes-that-wont-stay-independent.