Across six LLMs tested by ChronoCulture, predictions track the direction of a cultural shift better than its magnitude, and the magnitude gap widens as the real shift grows larger

LLM-evaluationcultural-awarenessbenchmarkworld-values-surveytemporal-reasoningarxivwasserstein-distance

ChronoCulture's headline finding — that model predictions for two different years tend to stay close together — can be decomposed further. The paper defines a "model shift" (the distance between a model's older-year and newer-year predictions, measured with Wasserstein distance) and a "human shift" (the same distance between the real older-year and newer-year survey distributions), then reports their ratio r = model-shift / human-shift. A ratio of 1 means the model's predicted change matches the observed change exactly; below 1 means the model underestimates how much actually changed.

For temporal pairs where a model's two predictions were not simply identical, the median value of this ratio stayed below 1 and declined further as the real survey shift increased — the more a culture actually changed, the smaller a fraction of that change the models' predictions captured. A separate, direction-only measure (whether the predicted change pointed the same way as the real change, ignoring size) improved as the real shift grew larger. So the models are not uniformly blind to change: they get better at guessing which way a culture is moving the more it moves, while getting worse, proportionally, at guessing how far.