Six LLMs tested as silicon-sampled survey respondents fail to cover real human respondents, and the failure survives every robustness check tried

llm-evaluationsilicon-samplingsurvey-methodssynthetic-dataarxiv

Kumagae & Kaneko (arXiv:2610.30900, submitted 22 Oct 2026) evaluate six LLMs — gpt-5.4-nano, gemini-3.1-flash-lite, Qwen3.5-9B, gemma-4-12B-it, gpt-oss-20b, and Llama-3.3-70B-Instruct — as silicon-sampled respondents to two real-world US surveys, the World Values Survey (WVS) wave 7 and the 2024 American National Election Studies (ANES) Time Series Cumulative Data File, scored on the paper's own respondent coverage gap (RCG) metric. Every model scored RCG above 1.0 on both surveys (WVS: 1.199–1.275; ANES: 1.277–1.370) — the nearest synthetic respondent to a given human respondent sits systematically farther away than another human respondent would. Under the paper's main, uncalibrated settings, all six LLMs scored worse than "Marginals," a non-LLM baseline that samples each answer independently from the human per-question marginal distributions and encodes no cross-question structure at all (Marginals: RCG 1.158 on WVS, 1.250 on ANES).

The authors then tried to close the gap, varying sampling temperature (0.1–1.5), reasoning effort, how much prior response history a model sees, model tier (adding gpt-5.4 and gemini-3.1-pro-preview), and the number of synthetic respondents generated per cell (up to 10,000). Some interventions brought individual models close to or below the Marginals baseline. None, in any combination, brought any model's RCG below "Copula" — a second non-LLM baseline modeling cross-question dependence with a Gaussian copula fit to the human data (Copula: RCG 1.041 on WVS, 1.091 on ANES). The undercoverage is not a tuning artifact; it survives every lever the paper pulls.

This is distinct from the separate finding that LLMs' unprompted survey-opinion distributions skew toward the most recent WVS wave (claim-liu-kaneko-chu-llm-opinion-alignment-strongest-with-most-recent-wvs-wave): that result concerns which population-level distribution a model lands near; this one concerns whether any individual synthetic respondent resembles any individual human respondent at all. See question-corroborate-silicon-sampling-rcg-single-source — this is one of three notes the vault currently rests on this single, three-day-old, uncorroborated preprint.