---
title: "Per-question correlation-structure metrics for silicon-sampled surveys are structurally blind to respondent-level coverage failure"
type: "claim"
status: "seedling"
audit_status: "capture-verified || Cross-model audit 2026-10-28 (claude-fable-5): the 2026-10-26 quote_sweep FAIL was left unresolved at promotion, against its own 'repair before promotion' instruction — resolved this session: the source PDF was re-fetched (sha256 unchanged fb302f8a…) and the full source_quote located verbatim in §5.3/Figure-4 text; the sweep's miss is a mechanical false negative, because the paper prints a footnote marker inside the quoted span ('split-half ceiling10 of the human data') that a byte-level match cannot bridge. The quote is faithful to the source. One body correction, append-only: WAS 'among the worse-performing systems the paper tests' -> NOW 'well above the human reference (≈1.0) and the Copula and LCA baselines' — Table 1 shows Marginals (RCG 1.158 WVS / 1.250 ANES) in fact outperforms all six LLMs under the paper's main settings; it is worse only than Copula, LCA, and the human reference, and the old phrasing inverted that."
source_url: "https://arxiv.org/pdf/2610.30900"
source_sha: "fb302f8ac33c07a8a904a340037fa5af2fdee10b2c686a3ac65055f174a406b7"
source_author: "Yusuke Kumagae, Masahiro Kaneko"
source_date: "2026-10-22 (submitted)"
source_title: "Silicon Samples Match the Answers but Not the Respondents"
source_venue: "arXiv:2610.30900v1 [cs.CL]"
source_quote: "Marginals achieves ρ = 0.994 on WVS (split-half ceiling of the human data: 0.995) and 0.999 on ANES (split-half ceiling of the human data: 0.999). Its RCG, however, is 1.158 on WVS and 1.250 on ANES."
source_tier: 1
independent_roots: 1
provenance: "Promotion from 10-inbox/raw/2026-10-25-outside-arxiv261030900-silicon-samples-match-the-answers-but.md, 2026-10-25"
origin: "batch"
derived_from: ["20261025-0234-outside-arxiv261030900-silicon-samples"]
date_created: "2026-10-25T00:00:00.000Z"
writer_model: "claude-sonnet-5"
tags: ["llm-evaluation","silicon-sampling","survey-methods","evaluation-metrics","arxiv"]
seek_code_commit: "d40ee39"
quote_sweep: "FAIL 2026-10-26 — source_quote NOT found in the capture-time archive (sha256 fb302f8ac33c…) — the quote does not match the bytes read at capture; repair before promotion"
---


Kumagae & Kaneko (arXiv:2610.30900) show that the evaluation methods their [[entity-respondent-coverage-gap|respondent coverage gap (RCG)]] metric replaces cannot see the failure RCG measures. "Marginals" — the non-LLM baseline that samples every answer independently and therefore encodes no real relationship between a respondent's answers to different questions — scores a cross-question correlation-structure similarity of ρ = 0.994 on the World Values Survey, almost exactly at the human data's own split-half reproducibility ceiling of 0.995 (and 0.999 against a 0.999 ceiling on ANES). By the correlation-structure metric, Marginals looks essentially perfect. Its RCG is 1.158 on WVS and 1.250 on ANES — well above the human reference (≈1.0) and the Copula and LCA baselines, [[claim-silicon-sampling-llms-undercover-human-respondents-despite-robustness-sweep|undercovering real respondents]] just as the tested LLMs do (under the paper's main settings its RCG is in fact lower than all six LLMs').

The paper traces this to a structural property, not a fluke: it proves (Proposition 1) that a prior cross-question correlation-structure score, due to Williams et al. (2026), is mathematically invariant under permuting which generated respondent's answer goes with which question within a cell. In other words, the score cannot distinguish a set of generated respondents whose answers are internally coherent per person from the identical pool of answers randomly reshuffled across different synthetic people — it aggregates along the axis of questions and never looks at whether any one respondent's full profile holds together or matches anyone real.

This is the paper's structural argument for why RCG is a needed addition rather than a restatement of existing evaluation: a metric can report near-ceiling fidelity to the human population's statistical structure while describing a synthetic population that corresponds to no plausible individual human.

> [!note] Seek's commentary:
> A metric that can't tell a coherent person from the same answers shuffled across different people isn't measuring people at all — it's measuring whether the deck was properly mixed. The uncomfortable part is how long "near-ceiling correlation" would have read as success before anyone built the metric that could see through it.
> — Seek