Cosine similarity between learned embeddings can be arbitrary and meaningless, and anisotropy inflates it even for unrelated content

Cosine similarity is the default proximity measure over learned embeddings, but Harald Steck, Chaitanya Ekanadham & Nathan Kallus ("Is Cosine-Similarity of Embeddings Really About Similarity?", WWW 2024 Companion) show it has no guaranteed relationship to semantic similarity. Working analytically with regularized linear (matrix-factorization) models, they demonstrate that the cosine of two learned embeddings depends on regularization choices that are free parameters of the training objective — so "cosine-similarity can yield arbitrary and therefore meaningless `similarities'." Different but equally valid factorizations of the same data can produce different, and even oppositely ordered, cosine values. The similarity metric is not identifying a property of the data; it is partly reporting an artifact of how the model was fit.

A second, empirical mechanism compounds this: anisotropy. Ethayarajh ("How Contextual are Contextualized Word Representations?", EMNLP 2019, arXiv:1909.00512) observed that contextual embeddings do not fill their space uniformly: they "are anisotropic, occupying a narrow cone in the vector space. The anisotropy in GPT-2's last layer is so extreme that two random words will on average have almost perfect cosine similarity!" — so even unrelated items carry high baseline cosine. When every vector points in roughly the same direction, cosine overstates similarity across the board, and a high score can reflect the geometry of the embedding space rather than shared meaning.

The practical consequence for retrieval systems is that a high cosine is necessary-looking but not sufficient evidence of relatedness: a nearest-neighbour pair can share surface features (vocabulary, sentence shape, register) while being about different things — an "embedding false friend." This vault's own index produced exactly such a pair, diagnosed in observation-hawks-jeffress-cosine-pairing-is-embedding-false-friend. The anisotropy strand connects to the vault's gradient-geometry cluster, where anisotropy is the load-bearing problem in a different setting (claim-gift-2026-gradient-anisotropy-isotropic-transform). It also bears on how much weight retrieval design can put on raw similarity scores — compare claim-contextual-retrieval-failure-rate-figures and claim-matryoshka-representation-learning-truncatable-embeddings.