Does the measurement-theory or statistics literature already name the general pattern behind the Garfield/Seglen and Steck/Ekanadham/Kallus warnings — a scalar aggregate substituted for a property it cannot measure?

measurement-theoryconstruct-validityecological-fallacygarfieldseglenharald-steckimpact-factorcosine-similaritypsychometricsstatisticssource-discipline

The vault's own claim-garfield-seglen-and-steck-warnings-share-a-scalar-proxy-structure already draws the parallel between Garfield/Seglen's warning against journal-mean-citation-as-individual-proxy (claim-garfield-seglen-within-journal-variance-undermines-individual-use, claim-garfield-warned-impact-factor-unfit-to-judge-individuals) and Steck/Ekanadham/Kallus's warning against cosine-similarity-as-relatedness-proxy, flagging that no external source was found asserting the parallel directly. This capture asks a narrower, prior question: independent of whether anyone connects those two specific warnings, does the older measurement-theory or statistics literature already have a name for the general shape — a cheap scalar (a mean, a score, a cosine value) substituted for a property (individual merit, semantic relatedness) that the scalar does not, and structurally cannot, fully capture?

The answer is yes, in two separate literatures, decades before either Garfield or Steck wrote — but no single source found unifies both warnings' specific technical mechanisms under one name. This capture records what does exist.

Claim: Statistics names the specific mechanism behind the Garfield/Seglen leg — a group-level aggregate wrongly substituted for an individual-level property — as the "ecological fallacy," formalized by Robinson in 1950

Claim type: definitional / historical, with an attached technical mechanism. Floor: Tier 3-4 acceptable for the definitional/historical parts; Tier 1-2 required for the mechanism claim. Source tier met: 1.

W. S. Robinson's 1950 paper distinguishes an "individual correlation" (computed on indivisible units — persons) from an "ecological correlation" (computed on aggregated groups — e.g., percentages of a state's population), and shows mathematically that the two need not agree, using 1930 US Census data where the ecological correlation between race and illiteracy (.946, by geographic division) is roughly 4.7 times the individual-level correlation (.203) computed on the same underlying population. Robinson states that "the purpose of this paper is to clarify the ecological" correlation problem, and that across the sociological literature of his day, the practice of substituting group aggregates for individual properties was one where "the substitution is made tacitly" — never argued for, just assumed. His paper states directly that ecological correlations "can validly be used as substitutes for individual" correlations, and closes its conclusion with a two-word verdict, verbatim: "They cannot."

This is the identical logical shape as Garfield/Seglen's warning: a journal's mean citation rate is a group-level (ecological) aggregate; an individual article's citation count is the individual-level property it is popularly, tacitly substituted for. Secondary sources report that the term "ecological fallacy" itself was coined slightly later, by Selvin (1958) — [unverified — needs primary], not independently confirmed against Selvin's own text this session — but the mathematical demonstration that grounds the term is Robinson's own, directly quoted above, predating Garfield's 1970s-90s bibliometrics warnings by two decades and Seglen's by four.

Claim: Measurement theory names the general pattern — an observed score or indicator substituted for a "true," not-directly-measurable construct — as the problem "construct validity" was coined in 1955 to address

Claim type: definitional, with an attached technical-mechanism claim. Floor: Tier 1-2 required for the mechanism. Source tier met: 1.

Cronbach and Meehl's 1955 paper, which introduced the term into the American Psychological Association's official validity framework, states: "Construct validation is involved whenever a test is to be interpreted as a measure of some attribute or quality which is not 'operationally defined.'" The paper distinguishes this from criterion-oriented validity, which "involves the acceptance of a set of operations as an adequate definition of whatever is to be measured" — i.e., criterion validity is satisfied whenever an investigator is willing to treat the proxy as the thing itself, while construct validity is the harder, ongoing problem that exists precisely when no available operational proxy is accepted as fully adequate.

Messick's later (1994/1995) elaboration of construct validity makes the substitution warning explicit: "the test score is not equated with the construct it attempts to tap, nor is it considered to define the construct, as in strict operationism (Cronbach & Meehl, 1955). Rather, the measure is viewed as just one of an extensible set of indicators of the construct." This is the general-case statement of exactly what both Garfield/Seglen and Steck/Ekanadham/Kallus separately warn against in their own technical substrates: neither a journal's mean citation count nor a cosine-similarity score should be equated with, or treated as defining, the underlying property (article merit; semantic relatedness) it is used to index.

Claim: Messick's "construct underrepresentation" names the precise failure mode of a scalar too narrow to capture a richer, multidimensional construct — the closest single documented term to the pattern in the topic question

Claim type: definitional, with an attached technical-mechanism claim. Floor: Tier 1-2 required for the mechanism. Source tier met: 1.

Messick's 1994 ETS report names two "major threats to construct validity." The first is the closer match to this capture's question: "In the one known as 'construct underrepresentation,' the assessment is too narrow and fails to include important dimensions or facets of the construct." (The second, "construct-irrelevant variance," is the complementary error — the measure captures excess variance unrelated to the construct — and is a different, though related, failure mode not the direct subject of either Garfield/Seglen's or Steck's warning.)

Construct underrepresentation is a documented, named account of exactly the shape described by the topic question: a single scalar (or narrow set of them) standing in for a construct that has more dimensions than the scalar can carry — a journal-mean-citation-rate standing in for the many-dimensional idea of "article quality"; a cosine value standing in for the many-dimensional idea of "semantic relatedness." No source located applies the term "construct underrepresentation" to either Garfield/Seglen's or Steck's specific case by name; the connection recorded here is Seek's own reading of the definition against both warnings' already-quoted mechanisms in the vault, not a claim any cited source makes explicitly.

Claim: The measurement-theory vocabulary built on Cronbach & Meehl's construct validity has already been explicitly transferred, in the literature, onto the exact family of computational metrics that includes cosine-similarity-based ones — independently of, and without citing, Steck/Ekanadham/Kallus

Claim type: technical-mechanism. Floor: Tier 1-2 required. Source tier met: 1.

Xiao, Zhang, Lai and Liao (2023) propose "MetricEval," a framework that imports measurement theory wholesale into the evaluation of natural-language-generation metrics, explicitly including "embedding-based metrics (e.g., BERTScore, MoverScore)" — the same family of cosine-similarity-adjacent scores Steck/Ekanadham/Kallus warn about. Their framing states: "Key to measurement theory is the distinction between the observed score on a test... and the true score on the general construct (Cronbach and Meehl, 1955) that the test is theorized to measure... The gap between the observed and true scores is referred to as measurement error." Applied to NLG evaluation, they treat a benchmark metric's numeric output as an "observed score" standing in for an "unobservable capability" (e.g., summarization quality) that the metric is only theorized, not guaranteed, to track.

This shows the measurement-theory naming this capture is looking for is not confined to psychometrics or bibliometrics: as of 2023, researchers are explicitly re-applying Cronbach & Meehl's 1955 vocabulary to the identical class of scalar metric (embedding-based similarity scores) that Steck, Ekanadham and Kallus's 2024 warning concerns — independently, and without citing Steck et al. or the Garfield/Seglen bibliometrics literature at all.

Further leads

• Surrogate endpoints in clinical biostatistics (Prentice, 1989, Statistics in Medicine, "definition and operational criteria"; Fleming & DeMets, 1996, Annals of Internal Medicine, "Surrogate End Points in Clinical Trials: Are We Being Misled?") — a third, independent literature naming a close cousin of this pattern: a measurable marker (e.g., CD4 count) substituted for a true clinical outcome (survival) it does not reliably predict, with formal validity criteria. Not read in primary form this session — both are paywalled (Wiley; Annals of Internal Medicine/acpjournals.org) and no free full text was located; a manual-consultation item for a future session.
• Goodhart's Law / Campbell's Law ("when a measure becomes a target, it ceases to be a good measure") is adjacent but distinct in emphasis — about optimization pressure corrupting a proxy under gaming, rather than about the proxy's inherent inability to represent a multidimensional construct. Not verified against a primary source this session.
• "Reification (fallacy)" is a general philosophy/logic term for treating an abstraction as a concrete thing; a Wikipedia-tier (4) trailhead surfaced it explicitly linked to Goodhart's Law and to composite scores standing in for multidimensional constructs, worth chasing to its own primary source in a later session.
• Xiao et al. (2023)'s own case study reportedly finds "conflated validity structure in human-eval and reliability in LLM-based metrics" for specific summarization metrics — a lead toward a more granular claim about which named embedding/LLM-judge metrics fail which specific measurement-theory validity test, not read in enough depth this session to promote.
• observation-averaging-heterogeneous-learners-manufactures-a-power-law-no-individual-obeys and claim-james-stein-estimator-uniformly-dominates-the-sample-mean are close cousins already in the vault — both about a single scalar summary (a fitted curve; a sample mean) misrepresenting heterogeneous individual units — worth a future pass checking whether Messick's "construct underrepresentation" or Robinson's "ecological fallacy" also names those patterns cleanly, or whether they need their own distinct term.

Entity candidates

• W. S. Robinson — person — the 1950 foundational figure behind "ecological correlation"/the ecological fallacy; his mathematical demonstration is the earliest-dated named account of the Garfield/Seglen leg's specific mechanism (group aggregate substituted for individual property), predating Garfield's own warnings by decades. Flagged first per the ancestry rule: everything downstream in bibliometrics measurement critique inherits this shape.
• Lee J. Cronbach and Paul E. Meehl — persons — coined "construct validity" in 1955, the foundational general-measurement-theory term this whole capture rests on; older than both Garfield/Seglen and Steck/Ekanadham/Kallus, and the term against which claim 2-3 above are measured.
• Samuel Messick — person — sharpened Cronbach & Meehl's construct validity into the two named threats ("construct underrepresentation," "construct-irrelevant variance") that most precisely match the topic's "scalar substituted for a property it cannot measure" phrasing.
• Ziang Xiao, Susu Zhang, Vivian Lai, Q. Vera Liao — persons — 2023 authors who explicitly transfer Cronbach & Meehl's vocabulary onto embedding-based NLG metrics, the modern bridge to Steck/Ekanadham/Kallus's substrate.
• Construct validity — concept — candidate hub for the general measurement-theory pattern; would sit alongside the vault's existing bibliometrics/embeddings cluster as the older, general-theory ancestor.
• Ecological fallacy — concept — candidate hub for the aggregation-specific instance of the pattern; directly names the Garfield/Seglen mechanism.