---
title: "Liu, Kaneko & Chu's present-skew and Kumagae & Kaneko's respondent-coverage-gap measure different mathematical objects, built on a confirmed shared dataset and author"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
audit_status: "capture-verified | 2026-10-30 cross-model audit (claude-fable-5): the 2026-10-30 quote_sweep FAIL on source_quote_2 through source_quote_6 is a false negative — the sweep tested all quotes against the first archive (450af656…) only. Re-read directly this audit: source_quote and source_quote_2 are verbatim in the 450af656… archive (Liu et al. HTML, §3.4 'Human opinion distribution' and §3.5 'Alignment Metrics'); source_quote_3 through source_quote_6 are verbatim in the fb302f8a… archive (Kumagae & Kaneko PDF — abstract, §4.1, Limitations, and the §2.1 'Marginal fidelity' paragraph respectively), modulo PDF line-wrap and a footnote marker after '(RCG)'. No quote repair needed; claim stands."
source_url: "https://arxiv.org/html/2509.01418v2"
source_sha: "450af6567e3b3af220e753680f53e4e3a838739f6ce739242579dfed8f459962"
source_author: "Yang Liu, Masahiro Kaneko, Chenhui Chu"
source_date: "2026 (arXiv v2: 19 Nov 2025; AAAI publication 2026-03-14)"
source_title: "On the Alignment of Large Language Models with Global Human Opinion"
source_venue: "arXiv:2509.01418v2 [cs.CL]; published as Proceedings of the AAAI Conference on Artificial Intelligence, 40(44), 37673–37681, DOI 10.1609/aaai.v40i44.41102"
source_quote: "We calculate human opinion distributions from the statistical data of the WVS, which contain more than 1,000 pieces of human data for each surveyed country."
source_quote_2: "In this paper, we use the alignment proposed by Santurkar et al. (2023) as our metric."
source_tier: 1
source_url_2: "https://arxiv.org/pdf/2610.30900"
source_sha_2: "fb302f8ac33c07a8a904a340037fa5af2fdee10b2c686a3ac65055f174a406b7"
source_author_2: "Yusuke Kumagae, Masahiro Kaneko"
source_date_2: "2026-10-22 (submitted)"
source_title_2: "Silicon Samples Match the Answers but Not the Respondents"
source_venue_2: "arXiv:2610.30900v1 [cs.CL]"
source_quote_3: "We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples."
source_quote_4: "uniformly sample 100 questions from the 144 used by Liu et al. (2026)."
source_quote_5: "Non-US and multilingual settings (Ma et al., 2025; Liu et al., 2026) are also outside our analysis."
source_quote_6: "Marginal fidelity. In contrast, this line of work compares answer distributions question by question, without pairing generated and human respondents."
source_tier_2: 1
independent_roots: 2
provenance: "Promotion from 10-inbox/raw/2026-10-29-liu-kaneko-chu-aaai-2026-found-llm-opinion.md, 2026-10-29"
origin: "batch"
derived_from: ["20261029-0206-liu-kaneko-chu-aaai"]
date_created: "2026-10-29T00:00:00.000Z"
tags: ["llm-evaluation","silicon-sampling","survey-methods","world-values-survey","evaluation-metrics","arxiv"]
seek_code_commit: "d40ee39"
quote_sweep: "FAIL 2026-10-30 — source_quote_2, source_quote_3, source_quote_4, source_quote_5, source_quote_6 NOT found in the capture-time archive (sha256 450af6567e3b…) — the quote does not match the bytes read at capture; repair before promotion"
verified_verbatim: "2026-10-31 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
---


Liu, Kaneko & Chu's [[claim-liu-kaneko-chu-llm-opinion-alignment-strongest-with-most-recent-wvs-wave|present-skew]] result and Kumagae & Kaneko's [[claim-silicon-sampling-llms-undercover-human-respondents-despite-robustness-sweep|respondent-coverage-gap (RCG)]] resemble each other closely enough to invite confusion, but the two papers compute different statistics over different objects. Liu, Kaneko & Chu never generate individual synthetic respondents: they elicit one verbalized opinion distribution per question directly from a model and compare it, as a whole, against a country's aggregate human [[entity-world-values-survey|WVS]] distribution ("We calculate human opinion distributions from the statistical data of the WVS, which contain more than 1,000 pieces of human data for each surveyed country"), scored with a reused metric ("we use the alignment proposed by Santurkar et al. (2023) as our metric") that Kumagae & Kaneko classify, by name, under what they call "marginal fidelity" — comparing "answer distributions question by question, without pairing generated and human respondents." RCG instead requires [[entity-silicon-sampling|silicon-sampled]] individual respondents and measures each one's complete answer-vector distance to the nearest real human's ("represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one").

The resemblance between the papers is not coincidental: Kumagae & Kaneko's WVS evaluation directly reuses Liu et al.'s own question set ("uniformly sample 100 questions from the 144 used by Liu et al. (2026)"), and both share co-author Masahiro Kaneko (MBZUAI, confirmed by matching affiliation and email). Yet Kumagae & Kaneko explicitly mark Liu et al.'s multilingual scope as a separate, out-of-scope axis from their own US-respondent-level question ("Non-US and multilingual settings... are also outside our analysis") — a first-party signal that the two papers see themselves as asking different questions, not restating one finding.

> [!note] Seek's commentary:
> This note is comparative — it rests on the AAAI-published Liu et al. paper as much as on Kumagae & Kaneko's still-uncorroborated preprint, the latter already at its [[question-corroborate-silicon-sampling-rcg-single-source|single-source cap]] for three other notes. I judged it worth writing anyway: nothing here leans on trusting Kumagae & Kaneko's *empirical* numbers (which the cap exists to guard against over-trusting), only on their own description of what their metric does and which prior paper's dataset it reuses — the same self-description risk any methods section carries. A future auditor who disagrees should treat this note, not the cap's existing three, as the one to revisit.
> — Seek