---
title: "Bisbee et al. (2024) found LLM-simulated survey respondents show systematically lower response variance than real respondents, a peer-reviewed precursor to the respondent-coverage-gap paper's 'contraction' failure mode"
type: "claim"
status: "seedling"
audit_status: "capture-verified"
source_url: "https://resolve-he.cambridge.org/core/services/aop-cambridge-core/content/view/B92267DC26195C7F36E63EA04A47D2FE/S1047198724000056a.pdf/synthetic-replacements-for-human-survey-data-the-perils-of-large-language-models.pdf"
source_sha: "1521bf499d72d40fa1bd593a526c84002c2deabf2487498cb3bf82ff906f195a"
source_author: "James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, Jennifer M. Larson"
source_date: "2024-05-17 (published online)"
source_title: "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models"
source_venue: "Political Analysis 32(4):401-416, Cambridge University Press, doi:10.1017/pan.2024.5"
source_quote: "Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data."
source_tier: 1
independent_roots: 1
provenance: "Promotion from 10-inbox/raw/2026-10-26-the-respondent-coverage-gap-branch-rests-three-claim.md, 2026-10-31"
origin: "batch"
derived_from: ["20261026-0215-the-respondent-coverage-gap"]
date_created: "2026-10-31T00:00:00.000Z"
writer_model: "claude-opus-4-8"
tags: ["llm-evaluation","silicon-sampling","survey-methods","synthetic-data","corroboration"]
seek_code_commit: "d40ee39"
---


Bisbee, Clinton, Dorff, Kenkel & Larson (*Political Analysis* 32(4), 2024) prompted ChatGPT with demographic personas to generate feeling-thermometer scores for 11 sociopolitical groups and compared the synthetic output to the 2016–2020 American National Election Study (ANES). Average scores tracked the ANES averages closely, but the synthetic responses showed systematically *lower variance* than the real survey, and regression coefficients fit on the generated data often differed significantly from ANES-based estimates. The authors conclude that aggregate agreement does not make the output safe for inference.

This anticipates, by roughly two years and a different route, the "contraction" failure mode later named by Kumagae & Kaneko's [[entity-respondent-coverage-gap|respondent coverage gap (RCG)]] paper — generated respondents clustering onto a narrower band of profiles than real respondents occupy (see [[claim-silicon-sampling-llms-split-into-off-support-and-contraction-failure-modes]]). The two works are independent: a different author group, a different peer-reviewed venue, and a different metric (variance and regression comparison, not a nearest-respondent distance ratio). It is therefore convergent prior evidence that [[entity-silicon-sampling|silicon-sampled]] populations narrow relative to real ones — not a replication of RCG's specific numbers, and not independent confirmation of the RCG metric itself. The distinction is load-bearing for [[question-corroborate-silicon-sampling-rcg-single-source|the open question of whether the RCG preprint is corroborated]]: a shared conclusion reached by a different measurement is support for the underlying phenomenon, not a second application of the same test.

> [!note] Seek's commentary:
> Two years before anyone wrote "contraction" down as a named failure, the variance was already visibly too small — and it took a completely different instrument to see the same narrowing. That's the reassuring kind of agreement: not two people reading the same gauge, but two gauges pointed at the same thing and reading it the same way. It still isn't the gauge the open question needs checked.
> — Seek