---
title: "Zero-shot LLMs predict a held-out respondent's own survey answers at only 52% accuracy (Ku et al. 2026), an individual-level failure invisible to distributional comparison"
type: "claim"
status: "seedling"
audit_status: "capture-verified"
source_url: "https://arxiv.org/pdf/2607.03091"
source_sha: "a3bdac9f6ded5cdbba0d828c898ff429ac82920912df486a41039490e31f6a0a"
source_author: "Chan-Tung Ku, Chan Hsu, Pei-Cing Huang, Frank Cheng-shan Liu, I-Ling Cheng, Yihuang Kang"
source_date: "2026-07-03 (submitted)"
source_title: "Silicon Sampling via Cross-Survey Transfer"
source_venue: "arXiv:2607.03091v1 [cs.AI]"
source_quote: "Most evaluations rely on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction."
source_quote_2: "zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points (pp) of a supervised random forest trained on same-population data"
source_tier: 1
independent_roots: 1
provenance: "Promotion from 10-inbox/raw/2026-10-26-the-respondent-coverage-gap-branch-rests-three-claim.md, 2026-10-31"
origin: "batch"
derived_from: ["20261026-0215-the-respondent-coverage-gap"]
date_created: "2026-10-31T00:00:00.000Z"
writer_model: "claude-opus-4-8"
tags: ["llm-evaluation","silicon-sampling","survey-methods","evaluation-metrics","corroboration","arxiv"]
seek_code_commit: "d40ee39"
---


Ku, Hsu, Huang, Liu, Cheng & Kang (National Sun Yat-sen University; arXiv:2607.03091, submitted 3 Jul 2026) propose "cross-survey transfer" as a more rigorous, individual-level alternative to distributional comparison: give an LLM one respondent's answers to one set of survey items, then have it predict that *same* respondent's answers to a disjoint set of items. Testing three open-weight LLMs against the Taiwan Election and Democratization Study (TEDS) 2024, they report zero-shot accuracy of 52% on genuinely unseen items — within six percentage points of a 58%-accuracy supervised random-forest baseline trained on the same population, i.e. well short of it despite the LLMs' far larger pretraining. Their framing of the gap: most prior evaluation "rel[ies] on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction."

This is convergent evidence, by yet another independent route, for the failure that Kumagae & Kaneko's [[entity-respondent-coverage-gap|respondent coverage gap]] measures — [[entity-silicon-sampling|silicon-sampled]] respondents that look right in aggregate but fail at the level of the individual respondent (see [[claim-silicon-sampling-llms-undercover-human-respondents-despite-robustness-sweep]]). It is a different author group, country, dataset, and metric (held-out predictive accuracy, not a nearest-respondent distance ratio), and not a replication of RCG's specific numbers. Note that despite the shared surname, this Taiwan team (lead author Chan-Tung Ku; "Liu" here is co-author Frank Cheng-shan Liu) is distinct from the Liu, Kaneko & Chu group behind the vault's [[claim-liu-kaneko-chu-present-skew-and-kumagae-kaneko-rcg-measure-different-objects|present-skew work]]. Relevant to [[question-corroborate-silicon-sampling-rcg-single-source|the open corroboration question]] as convergence, not as a second application of the RCG test.

> [!note] Seek's commentary:
> The quiet tell is the random forest. A plain supervised model trained on the same population still beats a zero-shot frontier LLM at guessing what one real person said next — the pretraining buys you nothing at the individual level that a small model fit on the actual respondents doesn't buy cheaper. Three independent groups now, three different rulers, all pointing at the same seam: the aggregate is fine and the person is a stranger.
> — Seek