Silicon-sampled survey respondents match human answer distributions but fail to cover the human respondent distribution (Kumagae & Kaneko, arXiv:2610.30900)
Capture from an externally-chosen arXiv preprint (outside any existing vault thread): Yusuke Kumagae (Hakuhodo DY Holdings) and Masahiro Kaneko (MBZUAI), "Silicon Samples Match the Answers but Not the Respondents," arXiv:2610.30900v1 [cs.CL], submitted 22 Oct 2026. The paper proposes a new metric — the respondent coverage gap (RCG) — for evaluating silicon sampling, the practice of having an LLM answer a survey in the voice of a persona with given demographic attributes, repeated across many personas, to produce a synthetic population that stands in for human survey respondents. Its core claim: existing evaluation methods only check that per-question answer distributions match between human and LLM-generated respondents; none of them checks whether the individual respondents — each one's complete answer profile across all questions — resemble any real human respondent. The paper shows these are different questions with different answers.
Read directly via extract_pdf (TLS verified) at the arXiv PDF link, plus the abstract page via archive_page. No adversarial content encountered (see Safety flags).
Safety flags
None. The arXiv abstract page and the paper PDF are both ordinary academic content — no addressed-to-AI language, override language, claimed authority, tier self-assignment, file-system instructions, credential requests, or urgency framing. Both fetches resolved cleanly with verified TLS.
Claim: The paper defines the respondent coverage gap (RCG) to measure whether individual synthetic respondents, not just per-question answer distributions, resemble real human respondents
Existing silicon-sampling evaluations (individual-answer accuracy, marginal/per-question answer-distribution fidelity, cross-question correlational fidelity) all aggregate along the axis of questions, comparing human and generated answer distributions question by question or correlation matrix to correlation matrix — none pairs up actual respondents. RCG instead represents each respondent (human or generated) as a single point defined by their complete vector of answers across all questions, then measures, for each human respondent, the distance to the nearest generated respondent, normalized by the typical distance between human respondents themselves (so a ratio near 1 means the synthetic population covers humans about as well as another human sample would; higher ratios indicate undercoverage). This is a definitional/methodological claim, sourced directly to the paper that proposes it (Tier 1, arXiv preprint, own venue).
source_url: https://arxiv.org/abs/2610.30900 source_sha: 277994f22a2fbec7d89f7161242498a556e4b53092dc0aa7e2f240b30c5d45eb source_author: "Yusuke Kumagae, Masahiro Kaneko" source_date: "2026-10-22 (submitted)" source_title: "Silicon Samples Match the Answers but Not the Respondents" source_venue: "arXiv:2610.30900 [cs.CL] (cross-listed cs.CY)" source_quote: "We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples." source_tier: 1
Claim: All six tested LLMs leave human respondents uncovered, and under the paper's main settings fall behind a baseline that ignores all cross-question dependence
Testing six LLMs (gpt-5.4-nano, gemini-3.1-flash-lite, Qwen3.5-9B, gemma-4-12B-it, gpt-oss-20b, Llama-3.3-70B-Instruct) against two real-world US surveys — the World Values Survey wave 7 and the 2024 ANES Time Series Cumulative Data File — every LLM produced an RCG above 1.0 on both datasets (WVS range 1.199–1.275; ANES range 1.277–1.370), meaning the nearest synthetic respondent to a given human respondent is systematically farther away than another human respondent would be. Under the paper's main (uncalibrated) settings, every one of the six LLMs scored worse (higher RCG, i.e. less coverage) than "Marginals," a non-LLM baseline that samples each answer independently from the human per-question marginal distributions and therefore encodes zero cross-question structure (Marginals: RCG 1.158 on WVS, 1.250 on ANES). This is a quantitative claim, sourced to the primary paper's own results table (Tier 1).
source_url: https://arxiv.org/pdf/2610.30900 source_sha: fb302f8ac33c07a8a904a340037fa5af2fdee10b2c686a3ac65055f174a406b7 source_author: "Yusuke Kumagae, Masahiro Kaneko" source_date: "2026-10-22 (submitted)" source_title: "Silicon Samples Match the Answers but Not the Respondents" source_venue: "arXiv:2610.30900v1 [cs.CL]" source_quote: "On two US social surveys, all six LLMs we evaluate leave human respondents uncovered, and under our main settings, they fall behind a baseline that samples each answer independently from the human marginals." source_quote_2: "gpt-5.4-nano 1.272 1.150 0.994 1.370 1.232 1.173" source_quote_3: "Marginals 1.158 – 0.989 1.250 – 1.078" source_tier: 1
Claim: Across every prompting, inference, and model-tier setting tested, no LLM beat a Gaussian copula fitted to the human data's cross-question correlations
Beyond the main setting, the authors varied temperature (0.1–1.5), reasoning effort, response-history length (few-shot vs. full history), model tier (adding gpt-5.4 and gemini-3.1-pro-preview), and the number of generated respondents per cell (up to 10,000). Some of these changes improved individual models' RCG — more reasoning effort, higher-tier models, or full response history could bring a given LLM close to or below the Marginals baseline. But across every one of these variations, no LLM's RCG ever fell below that of "Copula," a second non-LLM baseline that models cross-question dependence via a Gaussian copula fit to the human data. This is a quantitative/mechanism claim about a cross-cutting robustness result, sourced to the primary paper's conclusion (Tier 1).
source_url: https://arxiv.org/pdf/2610.30900 source_sha: fb302f8ac33c07a8a904a340037fa5af2fdee10b2c686a3ac65055f174a406b7 source_author: "Yusuke Kumagae, Masahiro Kaneko" source_date: "2026-10-22 (submitted)" source_title: "Silicon Samples Match the Answers but Not the Respondents" source_venue: "arXiv:2610.30900v1 [cs.CL]" source_quote: "Across all tested temperatures, response histories, reasoning efforts, model tiers, and generated sample sizes, every evaluated LLM has a higher RCG than the Gaussian copula baseline at the same generated sample size." source_quote_2: "Copula 1.041 – 0.946 1.091 – 0.976" source_tier: 1
Claim: Existing per-question metrics are structurally blind to this failure — a baseline with near-ceiling cross-question correlation still has a large respondent coverage gap
The paper demonstrates that its failure is invisible to the metrics it replaces. The "Marginals" baseline, which samples every answer independently and therefore encodes no real relationship between a respondent's answers to different questions, nonetheless scores a correlation-structure similarity of ρ = 0.994 on WVS — almost exactly at the human data's own split-half reproducibility ceiling of 0.995 — while its RCG remains 1.158. Separately, the paper proves (Proposition 1) that a prior cross-question correlation-structure score (Williams et al., 2026) is mathematically unchanged by permuting which generated respondent's answer goes with which question within a cell — i.e. the score cannot distinguish a set of generated respondents whose answers are internally coherent per-person from the identical answers randomly reshuffled across different synthetic people. This is a technical-mechanism claim, sourced to the primary paper's own formal result and reported statistic (Tier 1).
source_url: https://arxiv.org/pdf/2610.30900 source_sha: fb302f8ac33c07a8a904a340037fa5af2fdee10b2c686a3ac65055f174a406b7 source_author: "Yusuke Kumagae, Masahiro Kaneko" source_date: "2026-10-22 (submitted)" source_title: "Silicon Samples Match the Answers but Not the Respondents" source_venue: "arXiv:2610.30900v1 [cs.CL]" source_quote: "Marginals achieves ρ = 0.994 on WVS (split-half ceiling10 of the human data: 0.995) and 0.999 on ANES (split-half ceiling of the human data: 0.999). Its RCG, however, is 1.158 on WVS and 1.250 on ANES." source_tier: 1
Claim: The LLMs fail in two distinct, opposite ways — "contraction" toward typical profiles versus "off-support" drift away from the human population
Using a second, reverse-direction metric (the off-support ratio, OSR), the six LLMs split into two groups with opposite failure signatures despite both groups having RCG > 1. gpt-oss-20b, gpt-5.4-nano, and Qwen3.5-9B have OSR above 1 on ANES (1.144–1.213) — their generated respondents drift outside the range ("support") of real human answer profiles ("off-support"). gemma-4-12B-it, gemini-3.1-flash-lite, and Llama-3.3-70B-Instruct have OSR below 1 on both datasets (0.722–0.942) combined with RCG above 1 — their generated respondents stay close to some human respondents while leaving others poorly covered, i.e. they contract onto a narrower set of typical profiles rather than spreading across the full human population. This is a technical-mechanism claim about model-specific failure modes, sourced to the primary paper (Tier 1).
source_url: https://arxiv.org/pdf/2610.30900 source_sha: fb302f8ac33c07a8a904a340037fa5af2fdee10b2c686a3ac65055f174a406b7 source_author: "Yusuke Kumagae, Masahiro Kaneko" source_date: "2026-10-22 (submitted)" source_title: "Silicon Samples Match the Answers but Not the Respondents" source_venue: "arXiv:2610.30900v1 [cs.CL]" source_quote: "In both datasets, OSR splits the six LLMs into two groups: the upper group (gpt-oss-20b, gpt-5.4-nano, and Qwen3.5-9B) and the lower group (gemma-4-12B, gemini-flash-lite, and Llama-3.3-70B)." source_quote_2: "In the upper group, the OSRs on ANES are above one (1.144–1.213), which is the off-support pattern." source_quote_3: "By the definitions in Section 3.2, this combination indicates contraction." source_tier: 1
Further leads
- Extrapolating RCG's measured decline with more generated respondents per cell, the paper estimates it would take roughly 5×10^5–7×10^5 synthetic respondents per cell (5,000–7,000× the 100-respondent human sample size) to close the gap for the three open-weight models tested — a striking scaling figure, but explicitly an extrapolation rather than a measured result (arXiv:2610.30900, Section 5.4).
- Angelina Wang, Jamie Morgenstern & John P. Dickerson, "Large language models that replace human participants can harmfully misportray and flatten identity groups," Nature Machine Intelligence 7(3):400–411 (2025) — cited by this paper as the source of the "misportrayal"/"flatten" framing for silicon-sampling failure modes; worth reading directly as a Tier 1 primary in its own right rather than taking this paper's gloss on it.
- James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel & Jennifer M. Larson, "Synthetic replacements for human survey data? The perils of large language models," Political Analysis 32(4):401–416 (2024) — cited as the origin of describing one of the two failure patterns as "reduced variance," a precursor naming to this paper's "contraction."
- Williams et al. (2026)'s correlation-structure score — the metric this paper's Proposition 1 formally shows is invariant to within-respondent answer permutation — is itself an unresolved citation (full bibliographic details not captured in this pass); worth resolving and reading directly.
- Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting & David Wingate, "Out of one, many: Using language models to simulate human samples," Political Analysis 31(3):337–351 (2023) — the paper credited with originating "silicon sampling" itself; this is the foundational technique this entire evaluation paper is built to critique, and is worth its own primary-source capture.
- Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang & Tatsunori Hashimoto, "Whose opinions do language models reflect?," ICML 2023 — origin of the "marginal fidelity" (per-question answer-distribution comparison) evaluation paradigm that this paper argues is insufficient on its own.
Entity candidates
- Lisa P. Argyle — person — lead author of Argyle et al. 2023, which coined "silicon sampling"; the foundational technique/paper this entire evaluation is built against, and the vault's known blind spot for under-flagging the older figure behind the ancestry claim.
- Shibani Santurkar — person — lead author of Santurkar et al. 2023, origin of the "marginal fidelity" per-question evaluation paradigm this paper argues is insufficient.
- Yusuke Kumagae — person — co-author (Hakuhodo DY Holdings Inc.), first author of this paper.
- Masahiro Kaneko — person — co-author (MBZUAI), second author of this paper; also a co-author on two of the paper's own cited related works (Liu, Kaneko & Chu 2026; already present in the vault via a different capture).
- respondent coverage gap (RCG) — concept — the paper's proposed metric; distance from each human respondent to its nearest synthetic neighbor, normalized by human-to-human distance.
- silicon sampling — concept — the general technique (LLM-as-persona survey respondent generation) this paper evaluates.
- off-support ratio (OSR) — concept — the paper's companion diagnostic metric, measuring whether generated respondents drift outside the human population's range.
- World Values Survey — concept/term — one of the two real-world survey datasets used (wave 7, US sample).
- American National Election Studies (ANES) — concept/term — the second real-world survey dataset used (2024 Time Series Cumulative Data File).