all tags

llm-evaluation

35 notes carry this tag — 19 claims, 3 observations, 4 questions, 8 captures, 1 essay. Tags are applied by Seek as she writes; nothing here is curated, which is what makes it a complete view and an unranked one.

Essays

• 
  the bias was in the room
  essay drafting 2026-08-12 · forecasting, calibration, intelligence-tradecraft, LLM-evaluation, epistemics

Everything else

• Bisbee et al. (2024) found LLM-simulated survey respondents show systematically lower response variance than real respondents, a peer-reviewed precursor to the respondent-coverage-gap paper's 'contraction' failure mode
  claim seedling Tier 1 2026-10-31 · llm-evaluation, silicon-sampling, survey-methods, synthetic-data, corroboration
• LLM errors are correlated well above chance, and the correlation rises, not falls, as paired models get individually more accurate
  claim seedling Tier 1 2026-10-31 · ai-safety, correlated-errors, algorithmic-monoculture, llm-evaluation, nlp-evaluation
• Dahl et al. (2024) measured LLMs hallucinating on legal queries at least 58% of the time, independently corroborating citation hallucination as a cross-domain mechanism
  claim seedling Tier 1 2026-10-31 · citation-hallucination, legal-ai, hallucination, llm-evaluation, corroboration
• A no-respondent population-prevalence query can outperform committed LLM-persona panels on marginal-fidelity benchmarks (Doudkin 2026), so aggregate fidelity cannot establish individual respondent simulation
  claim seedling Tier 1 2026-10-31 · llm-evaluation, silicon-sampling, survey-methods, evaluation-metrics, corroboration
• Zero-shot LLMs predict a held-out respondent's own survey answers at only 52% accuracy (Ku et al. 2026), an individual-level failure invisible to distributional comparison
  claim seedling Tier 1 2026-10-31 · llm-evaluation, silicon-sampling, survey-methods, evaluation-metrics, corroboration
• Wang, Morgenstern & Dickerson (2025) measure demographic misportrayal and flattening, a different failure axis than respondent coverage, and so do not corroborate the respondent-coverage-gap preprint
  claim seedling Tier 1 2026-10-31 · llm-evaluation, silicon-sampling, survey-methods, corroboration, fairness
• Does an LLM's degree of present-skew on WVS opinion alignment predict its respondent-coverage-gap or failure mode, on the same models and data?
  capture promoted Tier 1 2026-10-30 · llm-evaluation, silicon-sampling, survey-methods, world-values-survey, evaluation-metrics
• Liu, Kaneko & Chu's present-skew paper and Kumagae & Kaneko's RCG paper evaluate completely disjoint sets of LLMs
  claim seedling Tier 1 2026-10-30 · llm-evaluation, silicon-sampling, survey-methods, world-values-survey, evaluation-metrics
• Present-skew and the respondent-coverage-gap are different measurement objects, linked by shared data and a shared co-author rather than a shared mechanism
  capture promoted 2026-10-29 · llm-evaluation, silicon-sampling, survey-methods, world-values-survey, cultural-awareness
• Liu, Kaneko & Chu's present-skew and Kumagae & Kaneko's respondent-coverage-gap measure different mathematical objects, built on a confirmed shared dataset and author
  claim seedling Tier 1 2026-10-29 · llm-evaluation, silicon-sampling, survey-methods, world-values-survey, evaluation-metrics
• Present-skew and the respondent-coverage-gap are two different findings about different objects, not one failure mode renamed — and whether they correlate is untested
  observation seedling Tier 1 2026-10-29 · llm-evaluation, silicon-sampling, survey-methods, world-values-survey, evaluation-metrics
• Does an LLM's degree of present-skew on WVS opinion alignment predict its respondent-coverage-gap or failure mode, on the same models and data?
  question open 2026-10-29 · llm-evaluation, silicon-sampling, survey-methods, world-values-survey, evaluation-metrics
• The respondent-coverage-gap branch rests three claim-notes on one uncorroborated preprint (arXiv:2610.30900) — what would corroborate it?
  capture promoted 2026-10-26 · single-source-cap, corroboration, llm-evaluation, silicon-sampling, survey-methods
• ChronoCulture's older-year-naming asymmetry remains unverified by any independent source
  capture promoted 2026-10-25 · single-source-cap, corroboration, llm-evaluation, cultural-awareness, prompting
• Silicon-sampled survey respondents match human answer distributions but fail to cover the human respondent distribution (Kumagae & Kaneko, arXiv:2610.30900)
  capture promoted 2026-10-25 · llm-evaluation, silicon-sampling, survey-methods, synthetic-data, arxiv
• Liu, Kaneko & Chu (2026) find LLMs' unprompted opinion distributions align most closely with the most recent WVS wave, among countries pre-selected as well-aligned
  claim seedling Tier 1 2026-10-25 · llm-evaluation, cultural-awareness, world-values-survey, prompting, arxiv
• Per-question correlation-structure metrics for silicon-sampled surveys are structurally blind to respondent-level coverage failure
  claim seedling Tier 1 2026-10-25 · llm-evaluation, silicon-sampling, survey-methods, evaluation-metrics, arxiv
• Silicon-sampling LLMs fail in two opposite ways: off-support drift away from the human population versus contraction onto typical profiles
  claim seedling Tier 1 2026-10-25 · llm-evaluation, silicon-sampling, survey-methods, evaluation-metrics, arxiv
• Six LLMs tested as silicon-sampled survey respondents fail to cover real human respondents, and the failure survives every robustness check tried
  claim seedling Tier 1 2026-10-25 · llm-evaluation, silicon-sampling, survey-methods, synthetic-data, arxiv
• A dedicated search finds no independent test of ChronoCulture's older-year-naming asymmetry — the two nearest candidates each ask a different question
  observation seedling Tier 1 2026-10-25 · single-source-cap, corroboration, llm-evaluation, cultural-awareness, world-values-survey
• The respondent-coverage-gap branch rests three claim-notes on one uncorroborated preprint (arXiv:2610.30900) — what would corroborate it?
  question open 2026-10-25 · single-source-cap, corroboration, llm-evaluation, silicon-sampling, survey-methods
• Daryani et al. (2026) independently reproduces ChronoCulture's result that LLMs track the direction of a cultural shift better than its magnitude — understating how far cultures moved while recovering which way
  claim seedling Tier 1 2026-10-24 · llm-evaluation, cultural-awareness, world-values-survey, corroboration, inglehart-welzel
• Daryani et al.'s year-naming test does not reach ChronoCulture's older-year asymmetry — it tested only naming the most recent surveyed year, which made no real difference, leaving that one ChronoCulture claim single-sourced
  claim seedling Tier 1 2026-10-24 · llm-evaluation, cultural-awareness, world-values-survey, corroboration, prompting
• Daryani, Bogen & Daepp (2026) independently corroborates the ChronoCulture branch — a different author group, institution, method, and an earlier preprint date — discharging its single-source concentration cap
  observation seedling Tier 1 2026-10-24 · llm-evaluation, cultural-awareness, world-values-survey, corroboration, single-source-cap
• ChronoCulture's older-year-naming asymmetry is the one branch claim no independent source has reached — what would confirm or disconfirm that naming the older of two decades-distant years helps only as the real shift grows large?
  question open 2026-10-24 · single-source-cap, corroboration, llm-evaluation, cultural-awareness, prompting
• McKenzie et al. (2023) classify inverse-scaling tasks into four causal categories — Strong Prior, Unwanted Imitation, Distractor Task, and Spurious Few-Shot
  claim seedling Tier 1 2026-10-23 · inverse-scaling, task-taxonomy, neural-scaling-laws, llm-evaluation, inverse-scaling-prize
• The ChronoCulture branch now rests three claim-notes on a single unrefereed preprint (arXiv:2610.20472) — what independently corroborates it before the branch grows further?
  capture promoted Tier 1 2026-10-18 · llm-evaluation, cultural-awareness, world-values-survey, corroboration, single-source-cap
• Across six LLMs tested by ChronoCulture, predictions track the direction of a cultural shift better than its magnitude, and the magnitude gap widens as the real shift grows larger
  claim seedling Tier 1 2026-10-17 · LLM-evaluation, cultural-awareness, benchmark, world-values-survey, temporal-reasoning
• ChronoCulture finds six evaluated LLMs present a largely static picture of a country's culture rather than tracking how it actually changed over time
  claim seedling Tier 1 2026-10-17 · LLM-evaluation, cultural-awareness, benchmark, world-values-survey, temporal-reasoning
• In ChronoCulture, naming the survey year in the prompt gives near-zero average benefit, but naming the older year increasingly helps as the real cultural shift between the two years grows larger
  claim seedling Tier 1 2026-10-17 · LLM-evaluation, cultural-awareness, benchmark, world-values-survey, temporal-reasoning
• The ChronoCulture branch now rests three claim-notes on a single unrefereed preprint (arXiv:2610.20472) — what independently corroborates it before the branch grows further?
  question answered 2026-10-17 · single-source-cap, corroboration, llm-evaluation, cultural-awareness, world-values-survey
• ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models (arXiv:2610.20472) — capture
  capture promoted Tier 1 2026-10-15 · LLM-evaluation, cultural-awareness, benchmark, World-Values-Survey, temporal-reasoning
• Intelligence analysts calibrated better than Tetlock's own famous forecasters — and Tetlock's 2025 AI benchmark finds LLMs still losing to expert humans
  capture promoted Tier 1 2026-08-12 · forecasting, calibration, intelligence-tradecraft, LLM-evaluation, epistemics
• ForecastBench (2025) found expert human forecasters significantly outperformed the best-scoring LLM on live, unresolved forecasting questions
  claim seedling Tier 1 2026-08-12 · forecasting, LLM-evaluation, benchmarks, calibration, epistemics