Silicon sampling

The practice of prompting an LLM to answer a survey question in the voice of a persona assigned specific demographic attributes, repeated across many personas, to produce a synthetic population that stands in for human survey respondents — cheaper and faster than fielding a real survey. The term and technique originate with Argyle, Busby, Fulda, Gubler, Rytting & Wingate, "Out of one, many: Using language models to simulate human samples" (Political Analysis 31(3), 2023); the vault has not yet read that paper directly.

Matters to this vault as the technique Kumagae & Kaneko's respondent coverage gap paper (arXiv:2610.30900, 2026) evaluates, finding that existing fidelity checks — which compare human and generated answer distributions — miss a failure at the level of individual synthetic respondents. Related but distinct from the separate finding that LLMs' unprompted survey-opinion outputs skew toward the most recent survey wave (claim-liu-kaneko-chu-llm-opinion-alignment-strongest-with-most-recent-wvs-wave) — that line of work tests opinion alignment generally, not the silicon-sampling persona-generation technique specifically, though both use World Values Survey data and share co-author Masahiro Kaneko.

References

• claim-silicon-sampling-llms-undercover-human-respondents-despite-robustness-sweep
• claim-per-question-metrics-blind-to-silicon-sampling-respondent-coverage-gap
• claim-silicon-sampling-llms-split-into-off-support-and-contraction-failure-modes
• claim-liu-kaneko-chu-present-skew-and-kumagae-kaneko-rcg-measure-different-objects
• observation-liu-kaneko-chu-present-skew-and-rcg-are-different-findings-not-renamed
• Captures: 10-inbox/raw/2026-10-25-outside-arxiv261030900-silicon-samples-match-the-answers-but.md, 10-inbox/raw/2026-10-29-liu-kaneko-chu-aaai-2026-found-llm-opinion.md

Log

• 2026-10-29: Settled, by direct comparison rather than inference, that silicon-sampling's respondent-coverage-gap and the separate present-skew finding (claim-liu-kaneko-chu-llm-opinion-alignment-strongest-with-most-recent-wvs-wave) are different measurement objects (aggregate distribution vs. individual-respondent profile), not one failure mode under two names — despite a confirmed shared co-author and a confirmed direct dataset-reuse link between the two papers. Whether the two actually correlate across models remains untested; see question-present-skew-rcg-correlation-untested.
• 2026-10-31: The practice has now accumulated multiple independent individual-level critiques, from author groups sharing no overlap with Kumagae & Kaneko: reduced response variance ("contraction") in ChatGPT personas vs. ANES (claim-bisbee-2024-llm-survey-respondents-show-reduced-variance, peer-reviewed, 2024); a no-respondent population query beating committed persona panels on marginal benchmarks (claim-doudkin-2026-no-respondent-query-beats-persona-panels, 2026); and zero-shot LLMs failing cross-survey individual-answer prediction (claim-ku-2026-cross-survey-transfer-individual-level-failure, 2026). A separate adjacent failure mode — demographic misportrayal/flattening — was distinguished from coverage and found not to corroborate it (claim-wang-2025-misportrayal-flattening-different-axis-than-coverage). All are convergent structural evidence that silicon-sampled populations fail at the individual level, not replications of the RCG metric. The technique's originator, entity-lisa-argyle, now has a hub.