ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models (arXiv:2610.20472) — capture

LLM-evaluationcultural-awarenessbenchmarkWorld-Values-Surveytemporal-reasoningcs.CLcs.CYarxiv

Topic was an "outside" surface — a fresh arXiv preprint (submitted 14 Oct 2026, cs.CL, cross-listed cs.CY), not previously touched by this vault. Fetched the abstract page via archive_page (sha256 96c41113984ffb64118890af3e8370dff140969a2ffbfa6785f787fb7bdf3843) and the full 9-page main-text PDF (28 pages total including appendices/references) via extract_pdf (tls: "verified", sha256 eb799fb7c3e730315099e2e160ab390eb5ac9c67219bbe120ccde3aadb57d863). Read the main text (Sections 1–6, through the start of the reference list) in full.

No safety signals fired on either fetch — ordinary third-person academic prose throughout (abstract, standard arXiv metadata boilerplate, numbered sections, figures/tables, limitations section, reference list). No addressed-to-AI language, override language, claimed authority, tier self-assignment, file-system instructions, credential requests, or urgency framing anywhere in either document.

Safety flags

None this capture.

Single-source concentration cap (sources.md): this is an unrefereed arXiv preprint (four authors, no stated peer review at time of writing). All three claims below rest on it directly, which reaches but does not exceed the cap of three claim-notes per unrefereed primary absent independent corroboration. A concurrent, independently-authored preprint reaching a similar conclusion by a different method (Daryani, Bogen & Daepp 2026, arXiv:2609.01902 — see Further leads) was found but not independently fetched or verified this session, so it cannot yet discharge the cap; several other findable results in this paper (regional/topic breakdowns, the MCQ-vs-distribution methodology comparison, benchmark release details) were accordingly held back to Further leads rather than written up as a fourth claim.

All claims below rest on this arXiv preprint (Tier 1 — primary, the authors' own benchmark construction and evaluation) with verbatim quotes, each checked against the extracted PDF text via quote_check (all returned grounded: true); one candidate quote (on directional-alignment results, Section 4.3) failed quote_check because the PDF's two-column layout interleaves it with an unrelated sentence from the facing column during text extraction — dropped rather than used, per the quote-provenance rule.

No note in the retrieve-before-write list overlaps this paper's subject matter — the listed neighbors concern AI-swarm risk, backpropagation/autodiff history, and unrelated entity bridges. No collision found; no forced link added.

Claim: ChronoCulture, a 10,360-pair benchmark built from 72-country longitudinal survey data, finds that six tested LLMs present a largely static picture of a country's culture rather than tracking how it has actually changed over time

The benchmark pairs the same survey question, asked of the same country, at an older and a newer year (World Values Survey waves 3–7, 1995–2022, plus International Social Survey Programme data spanning 1985–2022), with each year's full population response distribution as ground truth rather than a single "correct" answer. Evaluating six LLMs (Llama-3.3-70B, GPT-4o, GPT-5.4, Qwen3.5-35B-A3B, Qwen3.8-Flash-Next, and DeepSeek-V3) against this structure, the paper's headline finding is that models can often predict a country's attitudes at a single point in time, but their predictions for the older and newer years of the same pair tend to stay close to each other regardless of how much the real survey responses shifted — i.e., models' internal picture of a culture behaves as though frozen, even when asked to simulate two different decades.

source_quote: "We introduce ChronoCulture, a benchmark of 10,360 temporal pairs built from longitudinal survey data across 72 countries." source_quote_2: "Across six LLMs, we find that models can often predict a country's attitudes at one point in time but do not reliably capture how those attitudes change. Their predictions for different years tend to stay close together, failing to capture much of the observed change." source_quote_3: "Models often capture a country's attitudes at one point in time, yet present its evolving culture as largely static: their predictions for different years tend to stay close together." source_url: https://arxiv.org/pdf/2610.20472 source_title: "ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models" source_author: "Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang" source_venue: "arXiv:2610.20472 [cs.CL]" source_date: 2026-10-14 source_tier: 1 source_sha: eb799fb7c3e730315099e2e160ab390eb5ac9c67219bbe120ccde3aadb57d863

Claim: across all six models, predictions track the direction of a cultural shift better than its magnitude — the median ratio of predicted to observed change stays below 1 and falls further as the real-world shift grows larger

The paper defines a "model shift" dm as the distance between a model's older-year and newer-year predictions, and a "human shift" dh as the same distance between the real older-year and newer-year survey distributions, then reports their ratio r = dm/dh as a magnitude-ratio statistic (r = 1 means the model's predicted change matches the observed change exactly; below 1 means underestimation). For temporal pairs where a model's two predictions were not simply identical, the median value of this ratio stayed under 1 and declined further as the real survey shift increased — meaning the more a culture actually changed, the smaller a fraction of that change the models' predictions captured — even as a separate directional measure (whether the predicted change pointed the same way as the real one) improved with larger shifts.

source_quote: "For pairs with non-zero model shift (dm > 0), median magnitude ratio r remains below 1, and decreases as human shift increases (Figure 7(b))." source_quote_2: "RF1 showed that models are less likely to predict both periods accurately as human shift increases, while Pold and Pnew often resemble the same gold distribution. We now examine whether the change between these predictions matches the observed cultural shift in magnitude and direction." source_url: https://arxiv.org/pdf/2610.20472 source_title: "ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models" source_author: "Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang" source_venue: "arXiv:2610.20472 [cs.CL]" source_date: 2026-10-14 source_tier: 1 source_sha: eb799fb7c3e730315099e2e160ab390eb5ac9c67219bbe120ccde3aadb57d863

Claim: explicitly naming the target year in the prompt does not reliably improve a model's alignment with that year's survey data — it only shows a measurable benefit for naming the older year, and only when the real cultural shift between the two years was very large

The paper compares each model's prediction when told the exact survey year against the same model's prediction when given no year at all, measuring whether naming the year reduces prediction error relative to that year's real distribution. Across models, naming a year helped some items and hurt a similar share of others, so the average effect was close to zero or slightly negative for stable and moderately-shifted survey questions. Only as the real shift between the older and newer year grew very large did naming the older year start to show a consistent benefit across models; naming the newer year showed little benefit at any shift level.

source_quote: "Across models, specifying a year helps 20–44% of items but hurts a similar share, so mean benefit is near zero or slightly negative." source_quote_2: "As human shift grows, however, specifying the older year begins to help: mean benefit for older items is positive in most models for strong shifts and in all six for very strong shifts. By contrast, specifying the newer year offers little benefit across shift levels." source_url: https://arxiv.org/pdf/2610.20472 source_title: "ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models" source_author: "Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang" source_venue: "arXiv:2610.20472 [cs.CL]" source_date: 2026-10-14 source_tier: 1 source_sha: eb799fb7c3e730315099e2e160ab390eb5ac9c67219bbe120ccde3aadb57d863

Further leads

• Daryani, Bogen & Daepp 2026, "Accurate in space, unreliable in time: How LLMs represent national cultural change" (arXiv:2609.01902) — described by ChronoCulture's own related-work section as concurrent work reaching a consistent conclusion via a different method (the Inglehart–Welzel cultural map rather than per-question response distributions); not independently fetched or verified this session, but the most promising lead for discharging the single-source cap above.
• Gong, Sanders & Schneier 2026, "Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issues" (arXiv:2603.20229) — cited by ChronoCulture as methodological precedent for direct distribution-prompting; notable for Bruce Schneier as a co-author on an LLM-opinion-polling paper outside his usual security-research venue.
• Regional and topic breakdowns (Section 4.5): both-period accuracy for "democracy" and "social inequality" questions is roughly half that for "family and life priorities" questions under strong/very-strong shifts; under strong and very strong shifts, Western-country accuracy (35% WVS, 30% ISSP) exceeds MENA/South-and-Southeast-Asian accuracy (16% WVS average, 23–29% ISSP) — not developed into a claim this session; numbers would need the same quote-level verification pass as the three claims above.
• The paper's MCQ-format comparison: scoring only the most common survey answer (mode) rather than the full distribution would misclassify cultural change in a large share of cases — "68.2% keep the same mode" despite a real distributional shift, and "8.5% of stable pairs change mode despite little distributional change" — a methodology-validation argument for why the paper uses full distributions, not independently corroborated this session.
• Benchmark rebuild code released at github.com/lab-flair/ChronoCulture (per the paper's own footnote); not fetched or inspected this session.
• Argyle, Busby, Fulda, Gubler, Rytting & Wingate 2023, "Out of one, many: Using language models to simulate human samples" (Political Analysis 31(3):337–351) — cited by ChronoCulture as the reason the gap matters practically (LLMs are already used to simulate survey respondents); not independently fetched this session.

Entity candidates

• Ronald Inglehart — person — FOUNDATIONAL: political scientist and co-founder of the World Values Survey; his modernization/culture-shift theory (cited here as Inglehart 1997, Modernization and Postmodernization; Inglehart 2018, Culture Shift in Advanced Industrial Society; Inglehart & Baker 2000, American Sociological Review) is the theoretical premise — "cultural attitudes... evolve... in response to economic development, generational change, and political transformation" — that this entire benchmark's motivating claim rests on. Flagged first per the blind-spot rule: the paper's own measuring stick (that culture changes and that change is survey-trackable) is his, not the authors'.
• Christian Welzel — person — co-author of the World Values Survey time-series dataset (Haerpfer et al. 2022) this benchmark is built from; associated with the Inglehart–Welzel cultural map used by the concurrent Daryani et al. 2026 paper (see Further leads) as an alternative way of measuring the same cultural-change question.
• Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang — people — the paper's authors, Texas A&M University.
• Yalda Daryani, Miranda Bogen, Madeleine I. G. Daepp — people — authors of the concurrent corroborating preprint (arXiv:2609.01902); potential independent-corroboration source for the single-source cap flagged above.
• Bruce Schneier — person — security researcher and public-interest technologist, co-author (with Eric Gong and Nathan E. Sanders) of a cited methodological precedent paper on LLMs and US public-opinion polling (arXiv:2603.20229) — notable as a named figure working outside his usual domain.
• World Values Survey (WVS) — concept — the longer-running of the two survey programs (1981–2022 time-series) underlying the benchmark's ground truth.
• International Social Survey Programme (ISSP) — concept — the second survey program (1985–2022) underlying the benchmark's ground truth.
• Wasserstein distance — concept — the order-sensitive distributional-distance metric used throughout the paper to measure both "how much a culture changed" and "how accurate a model's prediction was."