Respondent coverage gap (RCG)

A metric proposed by Kumagae & Kaneko (arXiv:2610.30900, Oct 2026) for evaluating silicon-sampled survey data. It represents each respondent — human or LLM-generated — as a single point defined by their complete vector of answers across all questions, then measures, for each human respondent, the distance to the nearest generated respondent, normalized by the typical distance between human respondents themselves. A ratio near 1 means the synthetic population covers real humans about as well as another human sample would; a ratio above 1 means undercoverage.

Coined to fix a blind spot in prior evaluation methods, which compared human and generated answer distributions question-by-question or correlation-matrix-by-correlation-matrix without ever pairing up individual respondents. First seen in this vault 2026-10-25; not yet used by any paper outside the one that coined it.

References

• claim-silicon-sampling-llms-undercover-human-respondents-despite-robustness-sweep
• claim-per-question-metrics-blind-to-silicon-sampling-respondent-coverage-gap
• claim-silicon-sampling-llms-split-into-off-support-and-contraction-failure-modes
• claim-liu-kaneko-chu-and-kumagae-kaneko-test-disjoint-model-sets
• Captures: 10-inbox/raw/2026-10-25-outside-arxiv261030900-silicon-samples-match-the-answers-but.md, 10-inbox/raw/2026-10-30-does-an-llms-degree-of-present-skew-on.md

Log

• 2026-10-30: Two further facts from the same preprint, not yet promoted to their own claim-notes (the branch is at the single-source cap): across the six evaluated LLMs, RCG "shows no consistent ordering by model family, weight availability (proprietary vs. open-weight), or model size"; and a non-LLM Gaussian-copula baseline outperforms every evaluated LLM on RCG on both surveys (Copula RCG 1.041 WVS / 1.091 ANES vs. 1.199-1.275 WVS / 1.277-1.370 ANES for the LLMs). Both pending independent corroboration.
• 2026-10-31: A dedicated corroboration search (promotion of the 2026-10-26 follow-up capture) found NO source applying RCG, OSR, or an equivalent respondent-level coverage metric to independently reproduce this preprint's specific numbers — so the single-source cap on the preprint's empirical findings remains undischarged. It did surface convergent structural evidence, from independent author groups and by different metrics, that silicon-sampled populations fail at the individual level: claim-bisbee-2024-llm-survey-respondents-show-reduced-variance (a peer-reviewed precursor to "contraction"), claim-doudkin-2026-no-respondent-query-beats-persona-panels, and claim-ku-2026-cross-survey-transfer-individual-level-failure. One cited related work was checked and found to address a different failure axis, not coverage: claim-wang-2025-misportrayal-flattening-different-axis-than-coverage. Convergence on the phenomenon is not a second application of the test; the cap governs the latter. See the question's 2026-10-31 progress note.