---
id: "20261029-0206-liu-kaneko-chu-aaai"
title: "Present-skew and the respondent-coverage-gap are different measurement objects, linked by shared data and a shared co-author rather than a shared mechanism"
type: "capture"
status: "promoted"
origin: "batch"
promoted_to: ["30-notes/claim-liu-kaneko-chu-present-skew-and-kumagae-kaneko-rcg-measure-different-objects.md","30-notes/observation-liu-kaneko-chu-present-skew-and-rcg-are-different-findings-not-renamed.md"]
not_promoted: ["Claim 1 (dataset reuse / explicit out-of-scope scoping between the two papers) — not a standalone note; folded as supporting context into claim-liu-kaneko-chu-present-skew-and-kumagae-kaneko-rcg-measure-different-objects.md. Thin alone, and standalone would have been a 4th note resting on the single-source-capped Kumagae & Kaneko preprint with no comparative grounding to offset it.","Claim 3 (Kumagae & Kaneko's 'marginal fidelity' classification and Proposition 1 formal proof) — not a standalone note. Substantially overlaps claim-per-question-metrics-blind-to-silicon-sampling-respondent-coverage-gap.md (already in the vault), which demonstrates the same structural blindness via the Marginals baseline and Proposition 1; the one new fact (that Liu et al.'s own metric is explicitly named under this blind category) was folded into claim-liu-kaneko-chu-present-skew-and-kumagae-kaneko-rcg-measure-different-objects.md as one sentence, with a link to the existing note for the full numeric demonstration rather than re-deriving it.","Entity: Shibani Santurkar — declined a hub. Lead author of the foundational metric both papers measure against, but her own paper has not been read directly in this vault (only characterized secondhand by both papers under study); stays a mention per the vault's standing practice of not hubbing on secondhand characterization of an unread primary.","Entity: Esin Durmus — declined a hub, same reasoning as Santurkar (unread primary, secondhand characterization only, single paper).","Entity: Williams et al. (2026) — declined. Full citation still unresolved (surname and year only); 'when unsure, don't promote' per the entity spec. Already tracked as an open item in question-corroborate-silicon-sampling-rcg-single-source.md and repeated here as a further lead, not re-flagged separately.","Entity: Masahiro Kaneko — declined a hub again, consistent with the 2026-10-25 ruling already on record (seek-flags.md): two co-authored papers is stronger recurrence than a single joint paper, but this capture still doesn't treat him as a subject or actor in his own right, only confirms his identity is the same person across both bylines.","Entity: Hakuhodo DY Holdings Inc. — declined an org hub. Merely employs one author (Kumagae); does not itself act in either paper's argument; single mention, single cluster. Fails the org test's stricter bar.","Further leads (Santurkar et al. 2023 and Durmus et al. 2024 primaries unread; Williams et al. 2026 citation unresolved; Kumagae's Hakuhodo affiliation as a commercial-research angle) — left as leads in this capture, not promoted; none is load-bearing for a kept claim."]
writer_model: "claude-sonnet-5"
date_created: "2026-10-29T00:00:00.000Z"
provenance: "Batch research run, 2026-10-29"
derived_from: []
tags: ["llm-evaluation","silicon-sampling","survey-methods","world-values-survey","cultural-awareness","evaluation-metrics","arxiv"]
seek_code_commit: "d40ee39"
---


This capture answers a reflection hook: Liu, Kaneko & Chu (AAAI 2026) found that LLMs' unprompted opinion distributions align most strongly with the most recent World Values Survey (WVS) wave ("present-skew"; already in the vault as [[claim-liu-kaneko-chu-llm-opinion-alignment-strongest-with-most-recent-wvs-wave]]). Kumagae & Kaneko, read the same week, find LLM-generated silicon-sampled survey respondents match population-level WVS statistics while failing to cover which actual individual respondent they stand in for ("respondent-coverage-gap" / RCG; already in the vault as [[claim-silicon-sampling-llms-undercover-human-respondents-despite-robustness-sweep]], [[claim-per-question-metrics-blind-to-silicon-sampling-respondent-coverage-gap]], [[claim-silicon-sampling-llms-split-into-off-support-and-contraction-failure-modes]]). The question: is this one failure mode under two names, or two genuinely different findings that only resemble each other because both use WVS data and share a co-author (Masahiro Kaneko, MBZUAI, confirmed identical in both papers' own bylines/emails)?

Both primary sources were re-fetched directly this session: Kumagae & Kaneko's PDF via `extract_pdf` (TLS verified, sha256 `fb302f8a…`, matching the existing claim-notes) plus its arXiv abstract page via `archive_page` (sha256 `277994f2…`), and Liu, Kaneko & Chu's HTML arXiv page via `archive_page` (sha256 `450af656…`, matching the existing claim-note). No adversarial content encountered on either — both are ordinary academic text with no addressed-to-AI language, override language, claimed authority, tier self-assignment, file-system instructions, credential requests, or urgency framing.

**Flag for promotion:** Claims 2 and 3 below draw load-bearing quotes from Kumagae & Kaneko (arXiv:2610.30900), which already has three claim-notes resting on it per sources.md's single-source concentration cap (see [[question-corroborate-silicon-sampling-rcg-single-source]]). Both claims are comparative — they also rest on Liu, Kaneko & Chu (AAAI-published, i.e. peer-reviewed, not itself capped) — but promotion should weigh whether that joint grounding exempts them from the cap or whether they should route to the existing question note instead of becoming new standalone claim-notes.

## Claim: Kumagae & Kaneko's paper is built directly on Liu, Kaneko & Chu's own WVS question set and explicitly scopes the two papers' questions as different

Kumagae & Kaneko's WVS evaluation reuses Liu, Kaneko & Chu's own question-filtering work rather than building an independent question set: "uniformly sample 100 questions from the 144 used by Liu et al. (2026)." Their Limitations section also explicitly marks Liu et al.'s scope as orthogonal to their own: "Non-US and multilingual settings (Ma et al., 2025; Liu et al., 2026) are also outside our analysis." The "Liu et al. (2026)" cited is confirmed by the bibliography entry to be the same AAAI paper: "Yang Liu, Masahiro Kaneko, and Chenhui Chu. 2026. On the alignment of large language models with global human opinion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 37673–37681." Both papers list "Masahiro Kaneko" at "MBZUAI" with the identical email address on their own title pages, confirming the shared co-author is the same person, not a coincidence of a common name. The connection between the two papers is therefore tighter than "both happen to use WVS data": one explicitly builds on the other's dataset construction while treating its topic (multilingual/global alignment) as a distinct, out-of-scope axis from its own (US-only respondent-level coverage) — a first-party signal that the two papers' own authors see themselves as asking different questions, not restating one finding.

source_url: "https://arxiv.org/pdf/2610.30900"
source_sha: "fb302f8ac33c07a8a904a340037fa5af2fdee10b2c686a3ac65055f174a406b7"
source_author: "Yusuke Kumagae, Masahiro Kaneko"
source_date: "2026-10-22 (submitted)"
source_title: "Silicon Samples Match the Answers but Not the Respondents"
source_venue: "arXiv:2610.30900v1 [cs.CL]"
source_quote: "uniformly sample 100 questions from the 144 used by Liu et al. (2026)"
source_quote_2: "Non-US and multilingual settings (Ma et al., 2025; Liu et al., 2026) are also outside our analysis."
source_quote_3: "Yang Liu, Masahiro Kaneko, and Chenhui Chu. 2026. On the alignment of large language models with global human opinion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 37673–37681."
source_tier: 1

## Claim: The two papers measure different mathematical objects — a single aggregate opinion distribution versus a population of individual answer profiles

Liu, Kaneko & Chu's main method (RQ1 and RQ3, including the present-skew result) never generates individual synthetic respondents at all: it elicits one "verbalized" opinion distribution directly from the model per question and compares it, as a whole distribution, against a country's or wave's aggregate human answer distribution, computed from "more than 1,000 pieces of human data for each surveyed country": "We calculate human opinion distributions from the statistical data of the WVS, which contain more than 1,000 pieces of human data for each surveyed country." Their alignment score is a direct reuse of a prior metric: "In this paper, we use the alignment proposed by Santurkar et al. (2023) as our metric." This is a comparison of one distribution to another distribution — no individual respondent, synthetic or human, enters the computation.

Kumagae & Kaneko's RCG instead requires generating many persona-conditioned synthetic respondents and measures, for each real human respondent's complete multi-question answer vector, the distance to the nearest individual synthetic respondent's vector: "We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples." The two statistics are computed over different objects — one over a single aggregate distribution compared across historical waves, the other over a point cloud of individual answer profiles compared respondent-to-respondent — regardless of how similar the underlying survey data looks.

source_url: "https://arxiv.org/html/2509.01418v2"
source_sha: "450af6567e3b3af220e753680f53e4e3a838739f6ce739242579dfed8f459962"
source_author: "Yang Liu, Masahiro Kaneko, Chenhui Chu"
source_date: "2026 (arXiv v2: 19 Nov 2025; AAAI publication 2026-03-14)"
source_title: "On the Alignment of Large Language Models with Global Human Opinion"
source_venue: "arXiv:2509.01418v2 [cs.CL]; published as Proceedings of the AAAI Conference on Artificial Intelligence, 40(44), 37673–37681, DOI 10.1609/aaai.v40i44.41102"
source_quote: "We calculate human opinion distributions from the statistical data of the WVS, which contain more than 1,000 pieces of human data for each surveyed country."
source_quote_2: "In this paper, we use the alignment proposed by Santurkar et al. (2023) as our metric."
source_tier: 1
source_url_2: "https://arxiv.org/abs/2610.30900"
source_sha_2: "277994f22a2fbec7d89f7161242498a556e4b53092dc0aa7e2f240b30c5d45eb"
source_quote_3: "We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples."

## Claim: Kumagae & Kaneko's own argument and formal proof show that the metric family Liu et al.'s "alignment" score belongs to is structurally blind to the coverage failure RCG targets

Kumagae & Kaneko's related-work section classifies exactly the metric type Liu et al. reuse (Santurkar et al. 2023's alignment score) under what they call "marginal fidelity": "Marginal fidelity. In contrast, this line of work compares answer distributions question by question, without pairing generated and human respondents." They state the structural consequence directly: "In all of these, the human reference is itself an aggregate distribution; no individual enters the comparison." Their own metric's point is explicitly that this blindness is not cosmetic: "RCG is not a repackaging of existing metrics: marginal fidelity does not separate generators that differ in RCG, and an aggregate correlation metric is saturated by a generator that samples answers independently across questions." They demonstrate this concretely: a baseline that samples every answer independently, with zero cross-question coherence per respondent, scores almost exactly at the human data's own split-half reproducibility ceiling on a correlation-structure metric while still failing badly on RCG: "Marginals achieves ρ = 0.994 on WVS (split-half ceiling10 of the human data: 0.995) and 0.999 on ANES (split-half ceiling of the human data: 0.999). Its RCG, however, is 1.158 on WVS and 1.250 on ANES." Because Liu et al.'s own metric is drawn from exactly the family Kumagae & Kaneko name and formally argue against, the paper that documents present-skew is, by the newer paper's own logic, using a metric type that could not have detected a respondent-coverage failure even if one were present in the same models.

source_url: "https://arxiv.org/pdf/2610.30900"
source_sha: "fb302f8ac33c07a8a904a340037fa5af2fdee10b2c686a3ac65055f174a406b7"
source_author: "Yusuke Kumagae, Masahiro Kaneko"
source_date: "2026-10-22 (submitted)"
source_title: "Silicon Samples Match the Answers but Not the Respondents"
source_venue: "arXiv:2610.30900v1 [cs.CL]"
source_quote: "Marginal fidelity. In contrast, this line of work compares answer distributions question by question, without pairing generated and human respondents."
source_quote_2: "In all of these, the human reference is itself an aggregate distribution; no individual enters the comparison."
source_quote_3: "RCG is not a repackaging of existing metrics: marginal fidelity does not separate generators that differ in RCG, and an aggregate correlation metric is saturated by a generator that samples answers independently across questions."
source_tier: 1

## Claim: "Present-skew" and "respondent-coverage-gap" are two different findings about different objects, not one failure mode under two names — and whether they correlate empirically is untested

Taken together, claims 1–3 answer the core question: present-skew (which historical WVS wave's *aggregate* country-level distribution an LLM's single opinion output most resembles) and the respondent-coverage-gap (whether *individual* synthetic respondents' full answer profiles land near any real individual's profile) are genuinely different findings, not the same failure wearing two names. They share a data source (WVS) and a co-author (Masahiro Kaneko) by direct citation and byline, not by coincidence — but the resemblance between the two papers' topics comes from that shared substrate and from Kumagae & Kaneko visibly building on Liu et al.'s prior work, not from the two papers measuring the same statistic. Kumagae & Kaneko's own structural argument (claim 3) goes further: the metric family present-skew is measured with is, by construction, incapable of seeing a respondent-coverage failure one way or the other.

What neither paper tests, and what remains genuinely open: whether a model's degree of present-skew (how strongly it defaults to the most recent wave) predicts anything about its RCG or failure mode (contraction vs. off-support, per [[claim-silicon-sampling-llms-split-into-off-support-and-contraction-failure-modes]]) on the same survey. No source found during this session runs both measurements on the same models and data. This specific cross-paper correlation is marked **[unverified — could not confirm or deny after search]**; it was not the question either paper set out to answer, and answering it would require a new study, not a re-reading of these two.

> [!note] Seek's commentary:
> The two papers rhyme hard enough that it's tempting to read them as one discovery told twice — same survey, same middle name on the author list, same word "respondent" doing different work in each title. But claim 3 is the one that settles it for me: Kumagae & Kaneko aren't just measuring something different, they're explicitly arguing that the family of metric Liu et al.'s "present-skew" result rests on *cannot* see what RCG sees, by construction, with a baseline and a proof to back it up. That's not two names for one thing. That's one paper building the instrument the other one's method structurally lacked — which is a more interesting relationship than "duplicate finding," and a more honest one than "unrelated."
> — Seek

## Further leads

- Whether present-skew and RCG correlate across models (does stronger wave-recency bias predict worse or better respondent coverage, or a particular failure signature) is untested by any source found this session — a concrete open experiment for a future paper, not yet run by either author group.
- Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang & Tatsunori Hashimoto, "Whose opinions do language models reflect?," ICML 2023 — the origin of the alignment metric Liu et al. directly reuse and Kumagae & Kaneko classify as "marginal fidelity"; not yet read directly in the vault, only characterized secondhand by both papers under study here.
- Esin Durmus et al., "Towards measuring the representation of subjective global opinions in language models" (First Conference on Language Modeling, 2024) — the other paper in Kumagae & Kaneko's "marginal fidelity" pairing; also not yet read directly.
- Williams et al. (2026)'s correlation-structure score, which Kumagae & Kaneko's Proposition 1 formally proves is permutation-invariant — full bibliographic citation still not resolved (same open item flagged in the 2026-10-25 capture on this preprint).
- Kumagae & Kaneko's first author, Yusuke Kumagae, is affiliated with Hakuhodo DY Holdings Inc. (a Japanese advertising/marketing holding company), not an academic institution — a detail suggesting commercial market-research motivation behind the respondent-coverage question that has not been explored in this capture.

## Entity candidates

- Shibani Santurkar — person — lead author of Santurkar et al. 2023 ("Whose opinions do language models reflect?"), the foundational metric both papers in this capture measure themselves against: Liu et al. reuse her alignment score directly, and Kumagae & Kaneko build their entire critique around the "marginal fidelity" category her paper exemplifies. The older, foundational figure this capture's core argument rests on — flagged first per the vault's known blind spot on under-flagging ancestry figures.
- Esin Durmus — person — co-author of Durmus et al. 2024, the second exemplar in Kumagae & Kaneko's "marginal fidelity" category; extended Santurkar's metric to 60+ countries, directly antecedent to the aggregate-comparison approach present-skew uses.
- Williams et al. (2026, full citation unresolved) — the correlation-structure score formally refuted by Kumagae & Kaneko's Proposition 1 (permutation invariance); foundational antecedent on the RCG side of this capture's argument, not yet identified beyond the surname and year.
- Masahiro Kaneko — person — the shared co-author (MBZUAI) at the center of this capture's own question; co-authors both Liu, Kaneko & Chu (2026) and Kumagae & Kaneko (2026), confirmed identical by matching name, affiliation, and email across both papers' bylines.
- Hakuhodo DY Holdings Inc. — organization — Yusuke Kumagae's affiliation; a commercial advertising/market-research holding company rather than a university, worth a hub if the vault starts tracking industry (vs. academic) authorship on LLM-evaluation papers.