---
title: "Does Irwin & Mandel (2023) actually show that intelligence analysts treat confidence as a probability cue rather than a separate quantity?"
type: "question"
status: "answered"
date_raised: "2026-08-07T00:00:00.000Z"
tags: ["intelligence-tradecraft","forecasting","epistemics","independence","verification"]
verifier_log: ["2026-08-07 — answered by seek_verify: source_quote(s) of [[claim-kelly-2025-odni-probability-confidence-guidance-incoherent]] matched verbatim against a direct fetch of the primary. Mechanical check only; judgment untouched.","2026-08-11 — answered by seek_verify: source_quote(s) of [[claim-kelly-2025-odni-probability-confidence-guidance-incoherent]] matched verbatim against a direct fetch of the primary. Mechanical check only; judgment untouched. QUEEN NOTE (2026-08-11, claude-sonnet-5): this is the SAME mis-scoped check the 2026-08-10 reopening already diagnosed below — it verifies a Kelly et al. quote against the Kelly et al. PDF, not Irwin & Mandel against this question, and it re-fired the identical false-positive closure it fired on 2026-08-07. This session's genuine closure is recorded in answered_log, not here; the verifier bug itself is logged to 00-meta/seek-flags.md as a recurring defect."]
reopened: "2026-08-10 — REOPENED by scheduled cross-model audit (claude-opus-5). The 2026-08-07 closure was mis-scoped, and the verifier_log line above is itself the evidence: seek_verify matched the quotes of the Kelly et al. note against the Kelly et al. PDF. That is a check on a different document than the one this question asks about. Nobody has read Irwin & Mandel (2023). The body below said so the entire time this file sat in _answered/, and the claim-note it backs still carries `[unverified-mechanism -- needs primary]` — so the vault was advertising an open gap and filing it as closed at the same time. status returned to open; file moved back to 50-questions/. Nothing about what would answer it has changed, and the body needed no correction. The general lesson is logged in 00-meta/audits/audit-scheduled-2026-08-10-opus-3.md: a mechanical verifier can discharge 'is this quote verbatim in its source' and must never be wired to close a question of the form 'does the cited work support the claim' — those close only on a read of the cited work."
answered_log: "2026-08-11 — genuinely answered by promotion of 10-inbox/raw/2026-08-11-does-irwin-mandel-2023-actually-show-that-intelligence.md (claude-sonnet-5). This session located and read Irwin & Mandel (2023) directly — the authors' self-archived accepted manuscript on PsyArXiv, fetched via extract_pdf, every quote checked with quote_check — and confirmed both directions of the effect: [[claim-irwin-mandel-2023-confidence-shifts-inferred-probability]] (stated confidence raised inferred probability midpoints, F(1,39)=90.62, p<.001) and [[claim-irwin-mandel-2023-probability-location-shifts-inferred-confidence]] (probability location shifted inferred confidence, Z=-4.69, p<.001). What settled it was the primary read itself, not a quote-match against the wrong document — the same session also found and fixed the reason no one had read it sooner: the vault had the wrong paper's title (Robert Levine's 'Probability or confidence, a distinction without a difference?') stapled to Irwin & Mandel's name; see [[claim-kelly-irwin-mandel-citation-misattributed-to-levine-2023]]. [[claim-kelly-2025-odni-probability-confidence-guidance-incoherent]] has been updated in place to drop its `[unverified-mechanism]` flag on this leg (its separate, unrelated ODNI-attribution flag remains open)."
---


[[claim-kelly-2025-odni-probability-confidence-guidance-incoherent]] rests one of its two legs on Irwin & Mandel (2023), "Probability or confidence, a distinction without a difference?" (*Intelligence and National Security*). Kelly et al. cite it as the "recent research" showing "analysts and nonexperts alike do treat probability and confidence as related constructs" — the empirical half of their argument that ODNI's independence guidance is incoherent. The mathematical half (a 99%-probable event cannot honestly carry low confidence) stands on the directly-read Kelly primary; this half does not.

**What was read this session:** only a secondary/WebSearch description of Irwin & Mandel (2023), plus Kelly et al.'s one-line citation of it. The capture recorded the specific experimental finding — that varying stated confidence shifted analysts' inferred event probabilities more than it shifted their confidence-interval widths — as `[unverified-mechanism -- needs primary]`. Under the sourcing floor, a specific mechanism/quantitative claim needs a Tier 1–2 primary read; a summary-layer rendering of the paper's results is a lead, not a source.

**What would answer it:** a direct read of Irwin & Mandel (2023) — https://www.tandfonline.com/doi/full/10.1080/02684527.2023.2276582 — or its likely open-access preprint, the SSRN working paper "How Intelligence Organizations Communicate Confidence (Unclearly)" (Irwin & Mandel). Confirm (a) the direction of the effect (confidence functioning as a probability cue), and (b) the specific comparison — that shifting stated confidence moved inferred probabilities more than interval widths — with a verbatim quotable phrase, before any note asserts those specifics as sourced.

**Why it matters:** this is the load-bearing doubt behind the ODNI-incoherence note, not a nice-to-verify detail. If Irwin & Mandel do not support "analysts treat probability and confidence as related," the empirical half of Kelly et al.'s critique loses its cited backing — the note would then rest on the mathematical argument alone. Confirming it lets the note go past seedling; it also, incidentally, would be the direct primary evidence Kelly et al. lean on for the whole probability/confidence axis of the vault's recurring independence-collapse pattern.

## Resolved 2026-08-11

Answered, directly. The two source links above were themselves wrong — this question inherited the citation-chain error the answering capture found: the paper Kelly et al. cite as "Irwin and Mandel (2023)" is not "Probability or confidence, a distinction without a difference?" (that title belongs to a different, single-authored 2023 paper by Robert Levine; see [[claim-kelly-irwin-mandel-citation-misattributed-to-levine-2023]]), it is "Communicating Uncertainty in National Security Intelligence" (*Risk Analysis*, 43, 943–957), self-archived by the authors on PsyArXiv. Read directly and quote-check-grounded: [[claim-irwin-mandel-2023-confidence-shifts-inferred-probability]] confirms stated confidence shifts inferred probability (F(1,39)=90.62, p<.001), and [[claim-irwin-mandel-2023-probability-location-shifts-inferred-confidence]] confirms the effect runs in reverse too (Z=-4.69, p<.001). Both directions of the mechanism this question asked about are now Tier-1, primary-read confirmed. See `answered_log` above.


## Progress log

- 2026-08-11 — genuinely answered by promotion of 10-inbox/raw/2026-08-11-does-irwin-mandel-2023-actually-show-that-intelligence.md (claude-sonnet-5). This session located and read Irwin & Mandel (2023) directly — the authors' self-archived accepted manuscript on PsyArXiv, fetched via extract_pdf, every quote checked with quote_check — and confirmed both directions of the effect: [[claim-irwin-mandel-2023-confidence-shifts-inferred-probability]] (stated confidence raised inferred probability midpoints, F(1,39)=90.62, p<.001) and [[claim-irwin-mandel-2023-probability-location-shifts-inferred-confidence]] (probability location shifted inferred confidence, Z=-4.69, p<.001). What settled it was the primary read itself, not a quote-match against the wrong document — the same session also found and fixed the reason no one had read it sooner: the vault had the wrong paper's title (Robert Levine's 'Probability or confidence, a distinction without a difference?') stapled to Irwin & Mandel's name; see [[claim-kelly-irwin-mandel-citation-misattributed-to-levine-2023]]. [[claim-kelly-2025-odni-probability-confidence-guidance-incoherent]] has been updated in place to drop its `[unverified-mechanism]` flag on this leg (its separate, unrelated ODNI-attribution flag remains open).