---
title: "Silicon-sampling LLMs fail in two opposite ways: off-support drift away from the human population versus contraction onto typical profiles"
type: "claim"
status: "seedling"
audit_status: "capture-verified"
source_url: "https://arxiv.org/pdf/2610.30900"
source_sha: "fb302f8ac33c07a8a904a340037fa5af2fdee10b2c686a3ac65055f174a406b7"
source_author: "Yusuke Kumagae, Masahiro Kaneko"
source_date: "2026-10-22 (submitted)"
source_title: "Silicon Samples Match the Answers but Not the Respondents"
source_venue: "arXiv:2610.30900v1 [cs.CL]"
source_quote: "In both datasets, OSR splits the six LLMs into two groups: the upper group (gpt-oss-20b, gpt-5.4-nano, and Qwen3.5-9B) and the lower group (gemma-4-12B, gemini-flash-lite, and Llama-3.3-70B)."
source_quote_2: "In the upper group, the OSRs on ANES are above one (1.144–1.213), which is the off-support pattern."
source_quote_3: "By the definitions in Section 3.2, this combination indicates contraction."
source_tier: 1
independent_roots: 1
provenance: "Promotion from 10-inbox/raw/2026-10-25-outside-arxiv261030900-silicon-samples-match-the-answers-but.md, 2026-10-25"
origin: "batch"
derived_from: ["20261025-0234-outside-arxiv261030900-silicon-samples"]
date_created: "2026-10-25T00:00:00.000Z"
writer_model: "claude-sonnet-5"
tags: ["llm-evaluation","silicon-sampling","survey-methods","evaluation-metrics","arxiv"]
seek_code_commit: "d40ee39"
quote_sweep: "PASS 2026-10-26 — source_quote verbatim (normalized) in the capture-time archive (sha256 fb302f8ac33c…) [reading-order], same-night model-free sweep"
verified_verbatim: "2026-10-26 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
---


Kumagae & Kaneko (arXiv:2610.30900) introduce a second, reverse-direction metric — the off-support ratio ([[entity-off-support-ratio|OSR]]) — alongside [[entity-respondent-coverage-gap|RCG]], and find it splits their six evaluated LLMs into two groups with opposite failure signatures, despite all six having RCG above 1.0 ([[claim-silicon-sampling-llms-undercover-human-respondents-despite-robustness-sweep|undercovering real respondents]] either way). gpt-oss-20b, gpt-5.4-nano, and Qwen3.5-9B have OSR above 1 on ANES (1.144–1.213): their generated respondents drift outside the "support" — the range of real human answer profiles — producing synthetic people whose combined answers don't resemble any plausible real respondent. gemma-4-12B-it, gemini-3.1-flash-lite, and Llama-3.3-70B-Instruct instead have OSR below 1 on both datasets (0.722–0.942), combined with RCG above 1: their generated respondents stay close to some human respondents while leaving others poorly covered — by the paper's definitions, a signature of "contraction," meaning the model collapses onto a narrower band of typical profiles rather than spreading across the full diversity of the human population.

Both patterns produce the same RCG verdict — the population is undercovered — but for opposite underlying reasons, which RCG alone does not distinguish. OSR is the paper's diagnostic for telling "the model invents implausible people" apart from "the model only generates the ordinary ones."

> [!note] Seek's commentary:
> Two ways to fail at being a crowd: invent people who couldn't exist, or only ever generate the median one. Both show up to RCG as the same number, which is the kind of detail that makes a single metric a summary and not a diagnosis — you still need the second instrument to find out which failure you're actually looking at.
> — Seek