Silicon-sampling LLMs fail in two opposite ways: off-support drift away from the human population versus contraction onto typical profiles

llm-evaluationsilicon-samplingsurvey-methodsevaluation-metricsarxiv

Kumagae & Kaneko (arXiv:2610.30900) introduce a second, reverse-direction metric — the off-support ratio (OSR) — alongside RCG, and find it splits their six evaluated LLMs into two groups with opposite failure signatures, despite all six having RCG above 1.0 (undercovering real respondents either way). gpt-oss-20b, gpt-5.4-nano, and Qwen3.5-9B have OSR above 1 on ANES (1.144–1.213): their generated respondents drift outside the "support" — the range of real human answer profiles — producing synthetic people whose combined answers don't resemble any plausible real respondent. gemma-4-12B-it, gemini-3.1-flash-lite, and Llama-3.3-70B-Instruct instead have OSR below 1 on both datasets (0.722–0.942), combined with RCG above 1: their generated respondents stay close to some human respondents while leaving others poorly covered — by the paper's definitions, a signature of "contraction," meaning the model collapses onto a narrower band of typical profiles rather than spreading across the full diversity of the human population.

Both patterns produce the same RCG verdict — the population is undercovered — but for opposite underlying reasons, which RCG alone does not distinguish. OSR is the paper's diagnostic for telling "the model invents implausible people" apart from "the model only generates the ordinary ones."