construct validity

A term coined by Cronbach and Meehl in 1955 for the measurement-theory problem that exists whenever a test score is interpreted as a measure of an attribute that is not "operationally defined" — i.e., no available proxy is accepted as fully adequate to the thing it's meant to measure. Messick later split its failure modes into "construct underrepresentation" (the measure is too narrow to capture the construct's dimensions) and "construct-irrelevant variance" (the measure captures variance unrelated to the construct).

Matters to this vault as the general-measurement-theory ancestor of a pattern that keeps recurring under different names in different fields: it is the ground under claim-garfield-seglen-and-steck-warnings-share-a-scalar-proxy-structure's comparison of journal-mean-citation-count and cosine-similarity warnings, and it was independently transferred onto embedding-based NLG metrics by Xiao et al. (2023) without any awareness of either bibliometrics or embedding-retrieval literature citing it directly. Related but distinct from the jingle fallacy (same name, different construct) and the ecological fallacy (an aggregation-specific instance of a construct-validity-shaped substitution).

References

• claim-cronbach-meehl-1955-construct-validity-names-general-substitution-pattern
• claim-messick-construct-underrepresentation-names-scalar-too-narrow-for-construct
• claim-xiao-et-al-2023-metriceval-transfers-measurement-theory-to-embedding-metrics