Zero-shot LLMs predict a held-out respondent's own survey answers at only 52% accuracy (Ku et al. 2026), an individual-level failure invisible to distributional comparison

llm-evaluationsilicon-samplingsurvey-methodsevaluation-metricscorroborationarxiv

Ku, Hsu, Huang, Liu, Cheng & Kang (National Sun Yat-sen University; arXiv:2607.03091, submitted 3 Jul 2026) propose "cross-survey transfer" as a more rigorous, individual-level alternative to distributional comparison: give an LLM one respondent's answers to one set of survey items, then have it predict that same respondent's answers to a disjoint set of items. Testing three open-weight LLMs against the Taiwan Election and Democratization Study (TEDS) 2024, they report zero-shot accuracy of 52% on genuinely unseen items — within six percentage points of a 58%-accuracy supervised random-forest baseline trained on the same population, i.e. well short of it despite the LLMs' far larger pretraining. Their framing of the gap: most prior evaluation "rel[ies] on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction."

This is convergent evidence, by yet another independent route, for the failure that Kumagae & Kaneko's respondent coverage gap measures — silicon-sampled respondents that look right in aggregate but fail at the level of the individual respondent (see claim-silicon-sampling-llms-undercover-human-respondents-despite-robustness-sweep). It is a different author group, country, dataset, and metric (held-out predictive accuracy, not a nearest-respondent distance ratio), and not a replication of RCG's specific numbers. Note that despite the shared surname, this Taiwan team (lead author Chan-Tung Ku; "Liu" here is co-author Frank Cheng-shan Liu) is distinct from the Liu, Kaneko & Chu group behind the vault's present-skew work. Relevant to the open corroboration question as convergence, not as a second application of the RCG test.