Mandel & Barnes (2014) found Canadian intelligence analysts were underconfident, not overconfident, and their real-world forecasts explained far more outcome variance than Tetlock's landmark study

forecastingcalibrationintelligence-tradecraftepistemicsdavid-r-mandeltetlock

Mandel & Barnes (2014, PNAS) scored 1,514 real strategic-intelligence forecasts made by 15 Canadian government intelligence analysts over six years. Judgment-and-decision research's textbook finding is that experts run overconfident; this study found the reverse: "miscalibration was mainly due to underconfidence such that analysts assigned more uncertainty than needed given their high level of discrimination." By the paper's own account, the analysts' forecasts explained 76% of outcome variance, against roughly 20% in Philip Tetlock's landmark Expert Political Judgment tournaments of professional political forecasters — a comparison the paper draws explicitly, stating its results "provide a stark comparison with Tetlock's (17) findings." (The 76%/~20% contrast is Mandel & Barnes's own framing of Tetlock's number, read from their Tier-1 primary; it is not an independent re-check of Tetlock's original figure against his own book.)

The bias runs deeper for the forecasts that matter most: the paper's abstract notes miscalibration worsened for harder and more important forecasts, in the same direction (more hedged than the evidence warranted), rather than flipping toward overconfidence as difficulty rose. Mandel & Barnes name the underlying S-shaped calibration-curve pattern underextremity bias.

This finding sits inside David R. Mandel's DRDC (Toronto) research program tracking intelligence analysts' forecasting accuracy over years-long horizons — a program the vault's own entity page flagged on 2026-08-07 as unexplored beyond citation. It is the population-specific reversal whose comparison figure comes from Tetlock's tournaments — and eleven years later Tetlock, not Mandel, is a co-author of the benchmark that measures expert humans against a very different kind of forecaster: see claim-forecastbench-2025-expert-humans-beat-top-llm-forecaster. Mandel has no role in that benchmark, and this note is not its human baseline.