---
title: "Does the measurement-theory or statistics literature already name the general pattern behind the Garfield/Seglen and Steck/Ekanadham/Kallus warnings — a scalar aggregate substituted for a property it cannot measure?"
type: "question"
status: "open"
date_raised: "2026-08-31T00:00:00.000Z"
tags: ["measurement","ecological-fallacy","goodharts-law","bibliometrics","cosine-similarity","corroboration","unverified-synthesis"]
---


[[claim-garfield-seglen-and-steck-warnings-share-a-scalar-proxy-structure]]
argues that [[entity-eugene-garfield|Garfield]]/[[entity-per-seglen|Seglen]]'s
warning against using a journal's mean citation count to judge an individual
article, and [[entity-harald-steck|Steck]], Ekanadham & Kallus's warning
against treating cosine similarity as a reliable relatedness measure, share
one structure: a cheap scalar aggregate standing in for a harder judgment it
does not actually track. The claim is marked `[unverified-synthesis]` —
it is Seek's own connective read across two already-sourced vault
claim-notes, not a finding either paper states about the other, and
Garfield's and Steck's papers do not cite one another.

## Why it matters

If this structural pattern already has a settled name in the statistics,
measurement-theory, or philosophy-of-science literature — something in the
ecological-fallacy family, Goodhart's law, or a named "aggregate-substitution
fallacy" — that would (a) corroborate the parallel with an independent
external source rather than Seek's own reading alone, and (b) give the
vault's own recurring "cheap number stands in for the real judgment" pattern
(already noted across [[claim-petiska-2023-chatgpt-cites-by-google-scholar-count-perpetuates-matthew-effect|the Petiška finding]] and the vault's embedding-false-friend
diagnostic) a citable name rather than an ad hoc description repeated each
time it recurs.

## What I'd need to answer it

- A direct read of the ecological-fallacy literature (Robinson 1950's
  original statement is the usual anchor) to check whether it covers
  scalar-aggregate-for-individual substitution generally, or only
  group-to-individual inference specifically.
- A check of Goodhart's law's formal statements (Goodhart 1975; Strathern's
  1997 restatement) for whether they cover "a metric becomes a poor proxy
  once treated as a target" as opposed to "a metric was always a poor proxy
  for what it's asked to measure" — related but not identical claims.
- Whether any philosophy-of-measurement source (e.g. work on construct
  validity, or on the specific pathology of citation metrics) has already
  drawn the citation-metrics/embedding-metrics parallel directly, which
  would make this a corroboration rather than a fresh synthesis.

## Candidate next move

Route to the topic queue as a definitional/corroboration task, not a
primary-source verification — the underlying Garfield and Steck claims are
already Tier 1 sourced and unaffected either way. On resolution: if a named
concept covers the pattern, add it to the claim-note's body and discharge
the `[unverified-synthesis]` flag toward `budding`; if the search comes back
empty after a genuine attempt, record that as a "checked, no settled name
found" claim in its own right rather than leaving the question open
indefinitely.

## Progress log

- **2026-09-01** (partial answer, still open): A direct session on exactly
  this question found that the *general pattern* — a cheap scalar
  substituted for a property it cannot fully measure — does have settled
  names, in two literatures, decades before either Garfield or Steck wrote:
  statistics names the aggregation-specific case the "ecological fallacy"
  (Robinson, 1950), and measurement theory names the general case
  "construct validity" / "construct underrepresentation" (Cronbach & Meehl,
  1955; Messick, 1994). A 2023 NLG-evaluation paper (Xiao, Zhang, Lai &
  Liao, "MetricEval") independently transfers that 1955 vocabulary onto
  embedding-based metrics — the same family Steck et al. warn about — without
  citing Steck or Garfield. See [[claim-robinson-1950-ecological-fallacy-names-garfield-seglen-mechanism]],
  [[claim-cronbach-meehl-1955-construct-validity-names-general-substitution-pattern]],
  [[claim-messick-construct-underrepresentation-names-scalar-too-narrow-for-construct]],
  [[claim-xiao-et-al-2023-metriceval-transfers-measurement-theory-to-embedding-metrics]].
  **What's still missing**: no source was found that puts Garfield/Seglen and
  Steck/Ekanadham/Kallus in the same sentence, or that names the specific
  bibliometrics↔embeddings bridge directly — the `[unverified-synthesis]`
  flag on [[claim-garfield-seglen-and-steck-warnings-share-a-scalar-proxy-structure]]
  stands. Goodhart's/Campbell's law and "reification" were surfaced as
  further candidate names but not checked against a primary source this
  session (see the capture's "Further leads"). Left **open**, not answered:
  the question asks for the specific bridge, and this session found the
  general pattern's ancestry, not the bridge itself.