ChronoCulture finds six evaluated LLMs present a largely static picture of a country's culture rather than tracking how it actually changed over time

LLM-evaluationcultural-awarenessbenchmarkworld-values-surveytemporal-reasoningarxiv

ChronoCulture is a benchmark of 10,360 temporal pairs built from longitudinal cross-national survey data: the same survey question, asked of the same country, at an older and a newer year, drawn from World Values Survey waves 3–7 (1995–2022) and International Social Survey Programme data (1985–2022). Rather than scoring a single "correct" answer, each pair's ground truth is the full population response distribution for that year, so a model's prediction is judged against how people in that country actually answered at each of the two moments.

Evaluating six LLMs (Llama-3.3-70B, GPT-4o, GPT-5.4, Qwen3.5-35B-A3B, Qwen3.8-Flash-Next, and DeepSeek-V3) against this structure, the paper's headline finding is that models can often predict a country's attitudes at a single point in time, but their predictions for the older and newer years of the same pair tend to converge regardless of how much the real survey responses shifted. The practical consequence: a model asked to simulate the same country's values in, say, 1995 and 2020 will often answer both prompts as though from one frozen snapshot, even when the underlying population visibly changed its mind.

This finding motivates a magnitude/direction decomposition of the same gap (claim-chronoculture-llm-predictions-track-direction-better-than-magnitude-of-cultural-shift) and a test of whether naming the target year in the prompt helps (claim-chronoculture-naming-older-survey-year-helps-only-as-real-shift-grows-larger). The benchmark's entire premise — that a nation's culture is something which evolves over time in a way a survey can track — rests on Ronald Inglehart's modernization/culture-shift theory, which the paper cites but does not itself argue for.