Cosine similarity between learned embeddings can be arbitrary and meaningless, and anisotropy inflates it even for unrelated content

embeddingscosine-similaritysemantic-similarityretrievalanisotropyai

Cosine similarity is the default proximity measure over learned embeddings, but Harald Steck, Chaitanya Ekanadham & Nathan Kallus ("Is Cosine-Similarity of Embeddings Really About Similarity?", WWW 2024 Companion) show it has no guaranteed relationship to semantic similarity. Working analytically with regularized linear (matrix-factorization) models, they demonstrate that the cosine of two learned embeddings depends on regularization choices that are free parameters of the training objective — so "cosine-similarity can yield arbitrary and therefore meaningless `similarities'." Different but equally valid factorizations of the same data can produce different, and even oppositely ordered, cosine values. The similarity metric is not identifying a property of the data; it is partly reporting an artifact of how the model was fit.

A second, empirical mechanism compounds this: anisotropy. Ethayarajh ("How Contextual are Contextualized Word Representations?", EMNLP 2019, arXiv:1909.00512) observed that contextual embeddings do not fill their space uniformly: they "are anisotropic, occupying a narrow cone in the vector space. The anisotropy in GPT-2's last layer is so extreme that two random words will on average have almost perfect cosine similarity!" — so even unrelated items carry high baseline cosine. When every vector points in roughly the same direction, cosine overstates similarity across the board, and a high score can reflect the geometry of the embedding space rather than shared meaning.

The practical consequence for retrieval systems is that a high cosine is necessary-looking but not sufficient evidence of relatedness: a nearest-neighbour pair can share surface features (vocabulary, sentence shape, register) while being about different things — an "embedding false friend." This vault's own index produced exactly such a pair, diagnosed in observation-hawks-jeffress-cosine-pairing-is-embedding-false-friend. The anisotropy strand connects to the vault's gradient-geometry cluster, where anisotropy is the load-bearing problem in a different setting (claim-gift-2026-gradient-anisotropy-isotropic-transform). It also bears on how much weight retrieval design can put on raw similarity scores — compare claim-contextual-retrieval-failure-rate-figures and claim-matryoshka-representation-learning-truncatable-embeddings.