Did the Chater/Oaksford camp reply to Brighton's 2006 cross-validation reanalysis of the take-the-best result — and does the reversal stand?
This capture continues the corroboration task routed by question-corroborate-brighton-2006-take-the-best-reanalysis-chater-oaksford-reply, itself raised against claim-brighton-2006-reanalysis-reversed-take-the-best-result-against-chater-oaksford. That claim-note already records, from the Gigerenzer & Brighton side, that Brighton's 2006 reanalysis reversed the verdict of Chater, Oaksford, Nakisa & Redington's 2003 "Fast, frugal, and rational" study by switching from in-sample fit to cross-validation. This capture does not re-state that claim; it reports what a direct search for (a) a reply from the Chater/Oaksford camp and (b) direct verification of Brighton's own 2006 primary document actually turned up.
Claim: No reply from Chater, Oaksford, Nakisa, or Redington to Brighton's 2006 reanalysis has been located
Claim type: historical (absence claim). A genuine multi-angle search — web search across the 2003 paper's citation trail, the 2009 Homo Heuristicus target article and its Topics in Cognitive Science venue, and Oaksford/Chater's subsequent bibliography (Bayesian Rationality, 2007; Cognition and Conditionals, 2010; later single- and co-authored papers through 2025) — found no rejoinder, commentary, or later paper in which Chater, Oaksford, Nakisa, or Redington address Brighton's cross-validation reanalysis or defend their original fit-then-test-on-the-whole-sample protocol. Chater and Oaksford's own later work (e.g. the 2009 Bayesian Rationality précis, already in the vault as claim-oaksford-chater-redefine-rationality-as-reasoning-about-uncertainty and claim-oaksford-chater-recast-wason-task-as-optimal-data-selection) continues to develop their Bayesian rational-analysis programme without revisiting the city-population/take-the-best result.
Per the operating spec's guidance that an honest "could not confirm" is a valid outcome: this is recorded as such. The core question — did the rational-analysis camp ever respond — is [unverified — could not confirm or deny after search]. Absence of a located reply is not evidence that no reply exists (search coverage of decades-old cognitive-science venues is imperfect), but it is evidence that no reply is prominent enough to surface through citation trails, the journal's own subsequent issues, or general web search.
Claim: The AAAI 2006 Brighton paper — the primary document underlying the reversal claim — does not itself contain the city-population reanalysis it is cited for
Claim type: technical-mechanism / provenance. Tier 1 (direct read of the primary document
via extract_pdf).
Gigerenzer & Brighton (2009) cite "Brighton, H. (2006). Robust inference with simple cognitive models. In C. Lebiere & R. Wray (Eds.), AAAI spring symposium: Cognitive science principles meet AI-hard problems (pp. 17–22)" as the source of the reversal, writing: "Yet, as Brighton (2006) showed in a reanalysis, Chater et al.'s method of fitting the models on the learning sample and then testing these models on the entire sample (including the learning sample) favored those models that overfit the data... When cross-validation was used... The predictive accuracy of take-the-best exceeded that of all rival models over the entire range of sample sizes (Fig. 2)."
A direct read of that exact AAAI paper (6 pages, matching the cited pp. 17–22) shows it does not contain the city-population task, does not name or cite Chater, Oaksford, Nakisa, or Redington anywhere in its text or twenty-nine-item reference list, and does not contain a figure matching the four-panel comparison (linear perceptron, nearest neighbor, C4.5/CART, and a greedy take-the-best variant) that Gigerenzer & Brighton (2009) describe as "Fig. 2." Instead, the AAAI paper's own cross-validation analysis covers eight differently-named environments: "Figure 1(A) depicts predictive accuracy, estimated using CV, of the six models in eight natural environments as function of sample size," specifically "Biodiversity," "ProfessorSalary," "Bolts," "Oxidants," "Obesity," "FuelConsumption," "Mortality," and "Zoo" — regression datasets unrelated to city population. The AAAI paper itself further attributes its underlying dataset to "a larger study (Brighton 2006)" — i.e., even within the AAAI paper, "Brighton 2006" appears to refer to more than one document (the AAAI paper's own bibliography lists a separate "Brighton, H. 2006. Ecologically rational inductive inference. Manuscript in preparation").
This means the specific city-population reversal figure that Gigerenzer & Brighton's 2009 account rests on has not been independently verified against a directly-read primary source in this capture — only against the AAAI conference paper, which does not contain it, and against Gigerenzer & Brighton's own secondary description of it. This sharpens, rather than resolves, the open flag on claim-brighton-2006-reanalysis-reversed-take-the-best-result-against-chater-oaksford: the one-sidedness the existing claim already flags is compounded by a citation that does not resolve to the analysis it is credited with.
Claim: The one located post-2009 published exchange in the same venue is unrelated to take-the-best or the Chater/Oaksford dispute
Claim type: historical. Tier 1 (direct read of both documents).
Topics in Cognitive Science did carry a published exchange responding to Gigerenzer & Brighton's 2009 article: Hilbig, B. E., & Richter, T. (2011), "Homo heuristicus Outnumbered: Comment on Gigerenzer and Brighton (2009)," and Brighton & Gigerenzer's reply, "Towards Competitive Instead of Biased Testing of Heuristics: A Reply to Hilbig and Richter (2011)" (both Topics in Cognitive Science 3, 2011). A direct read of the reply confirms its subject: "Our programmatic article on Homo heuristicus (Gigerenzer & Brighton, 2009) included a methodological section specifying three minimum criteria for testing heuristics... Using Richter and Späth's (2006) study on the recognition heuristic, we illustrated how violations of these criteria can lead to unsupported conclusions." The entire exchange concerns the recognition heuristic (whether people use it consistently, per Richter & Späth's 2006 data), not the take-the-best heuristic, the city-population task, or Chater, Oaksford, Nakisa, or Redington's 2003 study — none of whom are authors or cited disputants in this exchange. This closes off a plausible false lead: the only named post-publication rebuttal-and-reply pair attached to Homo Heuristicus is not the reply being sought.
Further leads
- Brighton & Gigerenzer, "How heuristics exploit uncertainty," in Todd, Gigerenzer & the ABC Research Group (eds.), Ecological Rationality: Intelligence in the World (Oxford, listed as "in press" in the 2009 reference list) — may contain the full city-population reanalysis in primary form; not yet fetched. (Source: reference list of Gigerenzer & Brighton 2009.)
- Chater, N., Oaksford, M., Nakisa, R., & Redington, M. (2003), "Fast, frugal, and rational: How rational norms explain behavior," Organizational Behavior and Human Decision Processes 90, 63–86 — the original study itself has not yet been read directly in this vault; only described via secondary sources and search-engine summaries. A PDF mirror was located at dectech.co.uk/publications/LinksNick/ReasoningAndDecisionMaking/ but not yet fetched.
- Brighton & Gigerenzer (2015), "The bias bias," Journal of Business Research — a later restatement of the take-the-best debate citing Chater et al. (2003); not yet checked for whether it revisits or extends the reversal claim. (Found via ScienceDirect/ResearchGate listings.)
- Newell, B. R., Weston, N. J., & Shanks, D. R. (2003), "Empirical tests of a fast and frugal heuristic: Not everyone 'takes-the-best,'" Organizational Behavior and Human Decision Processes 91, 82–96 — a third, independent empirical critique of take-the-best, distinct from both the Gigerenzer and Chater/Oaksford camps; unexplored angle on whether the take-the-best result "stands" more broadly. (Source: cited in Gigerenzer & Brighton 2009 reference list.)
- Oaksford, M. (2025), "Social Rationality and Human Reasoning: Logical Expressivism and the Flat Mind," Topics in Cognitive Science — a recent Oaksford piece touching on ecological rationality; not checked for any retrospective comment on the take-the-best dispute. (Found via Wiley Online Library listing.)
Entity candidates
- Nick Chater — person — co-author of the 2003 study Brighton's reanalysis is measured against; the older, foundational figure on the rational-analysis side of this specific empirical dispute; no entity page yet exists in the vault.
- Mike Oaksford — person — co-author of the 2003 study, same foundational role as Chater above; no entity page yet exists.
- Ramin Nakisa — person — co-author of the 2003 study; harder to find independent biographical material, but a named party to the original result being reanalyzed.
- Michael Redington — person — co-author of the 2003 study; same as above.
- Henry Brighton — person — author of the 2006 reanalysis and co-author (with Gigerenzer) of Homo Heuristicus and its 2011 methodological sequel; the central figure whose unreplied-to critique this whole question turns on.
- Daniel G. Goldstein — person — co-creator with Gigerenzer of the take-the-best heuristic itself (Gigerenzer & Goldstein, 1996), the still-earlier foundational work that both the 2003 study and the 2006 reanalysis are testing.
- cross-validation (model-selection criterion) — concept — the specific statistical method whose adoption (vs. fit-then-test-on-whole-sample) is reported to flip the take-the-best result; candidate for its own note given how load-bearing it is across this dispute.
- bias–variance dilemma — concept — the statistical framing Gigerenzer & Brighton use to explain why cross-validation reverses the verdict; recurs across multiple sources found this run.