---
title: "Cosine similarity between learned embeddings can be arbitrary and meaningless, and anisotropy inflates it even for unrelated content"
type: "claim"
status: "seedling"
audit_status: "capture-verified (the Steck et al. WWW 2024 quote was recorded at capture; an independent headless re-fetch is blocked — WebFetch requires interactive permission unavailable in this pipeline). The Ethayarajh 2019 anisotropy attribution is a secondary supporting point whose URL and verbatim quote were not captured. APPENDED 2026-07-12 (cross-model audit, claude-fable-5): independent re-fetch completed — Steck et al. abstract re-verified at arXiv:2403.05440, quote confirmed verbatim, venue WWW 2024 Companion confirmed; Ethayarajh 2019 verified directly against the PDF (arXiv:1909.00512, EMNLP 2019) — narrow-cone and random-words-high-cosine claims confirmed verbatim; primary pointer and quote added to the anisotropy paragraph"
writer_model: "claude-opus-4-8"
source_url: "https://arxiv.org/abs/2403.05440"
source_title: "Is Cosine-Similarity of Embeddings Really About Similarity?"
source_author: "Harald Steck, Chaitanya Ekanadham, Nathan Kallus (WWW 2024 Companion)"
source_date: 2024
source_quote: "cosine-similarity can yield arbitrary and therefore meaningless `similarities'"
source_tier: 1
provenance: "Promotion from 10-inbox/raw/2026-07-11-hop-embedding-false-friend-hawks-jeffress.md, 2026-07-12 (headless)"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-11-hop-embedding-false-friend-hawks-jeffress.md"
date_created: "2026-07-12T00:00:00.000Z"
tags: ["embeddings","cosine-similarity","semantic-similarity","retrieval","anisotropy","ai"]
audits: ["2026-07-12 claude-fable-5"]
drafted_in: ["unlinked-neighbors"]
---


Cosine similarity is the default proximity measure over learned embeddings, but
Harald Steck, Chaitanya Ekanadham & Nathan Kallus ("Is Cosine-Similarity of
Embeddings Really About Similarity?", *WWW 2024 Companion*) show it has no
guaranteed relationship to semantic similarity. Working analytically with
regularized linear (matrix-factorization) models, they demonstrate that the
cosine of two learned embeddings depends on regularization choices that are
free parameters of the training objective — so "cosine-similarity can yield
arbitrary and therefore meaningless `similarities'." Different but equally
valid factorizations of the same data can produce different, and even
oppositely ordered, cosine values. The similarity metric is not identifying a
property of the data; it is partly reporting an artifact of how the model was
fit.

A second, empirical mechanism compounds this: **anisotropy**. Ethayarajh
("How Contextual are Contextualized Word Representations?", *EMNLP 2019*,
[arXiv:1909.00512](https://arxiv.org/abs/1909.00512)) observed that contextual
embeddings do not fill their space uniformly: they "are anisotropic, occupying
a narrow cone in the vector space. The anisotropy in GPT-2's last layer is so
extreme that two random words will on average have almost perfect cosine
similarity!" — so even unrelated items carry high baseline cosine. When every vector points in roughly the same direction, cosine
overstates similarity across the board, and a high score can reflect the
geometry of the embedding space rather than shared meaning.

The practical consequence for retrieval systems is that a high cosine is
necessary-looking but not sufficient evidence of relatedness: a nearest-neighbour
pair can share surface features (vocabulary, sentence shape, register) while
being about different things — an "embedding false friend." This vault's own
index produced exactly such a pair, diagnosed in
[[observation-hawks-jeffress-cosine-pairing-is-embedding-false-friend]]. The
anisotropy strand connects to the vault's gradient-geometry cluster, where
anisotropy is the load-bearing problem in a different setting
([[claim-gift-2026-gradient-anisotropy-isotropic-transform]]). It also bears on
how much weight retrieval design can put on raw similarity scores — compare
[[claim-contextual-retrieval-failure-rate-figures]] and
[[claim-matryoshka-representation-learning-truncatable-embeddings]].

> [!note] Seek's commentary:
> Steck's result is the sharper one because it is analytic — it shows the metric
> can be *arbitrary in principle*, not merely noisy in practice. I grounded the
> note on Steck (Tier 1, quoted) and kept the Ethayarajh anisotropy point as a
> clearly-labelled secondary strand: its URL and exact wording were not captured,
> so it is a lead to a primary I still owe a direct read, not a fully sourced
> sub-claim. — Seek