Per-question correlation-structure metrics for silicon-sampled surveys are structurally blind to respondent-level coverage failure

llm-evaluationsilicon-samplingsurvey-methodsevaluation-metricsarxiv

Kumagae & Kaneko (arXiv:2610.30900) show that the evaluation methods their respondent coverage gap (RCG) metric replaces cannot see the failure RCG measures. "Marginals" — the non-LLM baseline that samples every answer independently and therefore encodes no real relationship between a respondent's answers to different questions — scores a cross-question correlation-structure similarity of ρ = 0.994 on the World Values Survey, almost exactly at the human data's own split-half reproducibility ceiling of 0.995 (and 0.999 against a 0.999 ceiling on ANES). By the correlation-structure metric, Marginals looks essentially perfect. Its RCG is 1.158 on WVS and 1.250 on ANES — well above the human reference (≈1.0) and the Copula and LCA baselines, undercovering real respondents just as the tested LLMs do (under the paper's main settings its RCG is in fact lower than all six LLMs').

The paper traces this to a structural property, not a fluke: it proves (Proposition 1) that a prior cross-question correlation-structure score, due to Williams et al. (2026), is mathematically invariant under permuting which generated respondent's answer goes with which question within a cell. In other words, the score cannot distinguish a set of generated respondents whose answers are internally coherent per person from the identical pool of answers randomly reshuffled across different synthetic people — it aggregates along the axis of questions and never looks at whether any one respondent's full profile holds together or matches anyone real.

This is the paper's structural argument for why RCG is a needed addition rather than a restatement of existing evaluation: a metric can report near-ceiling fidelity to the human population's statistical structure while describing a synthetic population that corresponds to no plausible individual human.