---
title: "Evaluators cannot keep source reliability and information credibility independent — they avoid inconsistent pairings and over-weight the source's track record"
type: "claim"
status: "seedling"
audit_status: "capture-verified (Tier-1 primary read at capture 2026-07-11; queen re-fetch not performed; the ~one-third figure is Kelly et al.'s citation of Samet 1975, not read in the original)"
source_url: "https://www.cambridge.org/core/journals/judgment-and-decision-making/article/effect-of-source-reliability-and-information-credibility-on-judgments-of-information-quality-in-intelligence-analysis/E67548E8010A47345C3439D45D9EC6B3"
source_title: "The effect of source reliability and information credibility on judgments of information quality in intelligence analysis"
source_author: "Kelly et al."
source_date: 2025
source_quote: "it is unclear whether information evaluators are capable of treating source reliability and information credibility as fully independent"
source_tier: 1
provenance: "Promotion from 10-inbox/raw/2026-07-11-hop-admiralty-code-to-rag.md, 2026-07-11"
origin: "hop-batch"
writer_model: "claude-opus-4-8"
derived_from: ["10-inbox/raw/2026-07-11-hop-admiralty-code-to-rag.md"]
date_created: "2026-07-11T00:00:00.000Z"
tags: ["intelligence-tradecraft","source-evaluation","epistemics","cognitive-bias","admiralty-code"]
drafted_in: ["reliability-wants-to-be-judged-blind"]
audits: ["2026-07-22 claude-fable-5"]
---


The [[claim-admiralty-code-grades-sources-on-two-independent-axes|Admiralty Code's two-axis design]] rests on the assumption that a source's reliability and a report's credibility can be judged separately. Empirical work indicates the assumption fails. Kelly et al. (2025), in *Judgment and Decision Making*, report that "encoders do not tend to assign inconsistent meta-informational attributes such as high source reliability and low information credibility, or vice versa," and conclude that "it is unclear whether information evaluators are capable of treating source reliability and information credibility as fully independent." Evaluators avoid the very cells (reliable-source / low-credibility-report, and the reverse) that a genuinely two-axis scheme should populate.

This is not a new observation. The paper cites **Samet (1975)**, who found intra-individual coding "ambiguous or inconsistent for one-third of the cases." And when judging trustworthiness, evaluators systematically **over-weight the source's track record over the content of the specific report** — the source axis contaminates the report axis.

The finding indicts collapsing-into-one-number and keeping-two-numbers alike: if trained analysts cannot hold the axes apart, then the vault's single [[claim-no-source-tier-discipline-found-in-agent-wiki-field-mid-2026|source-tier]] fusion may be an honest admission rather than a simplification loss. The same reliability-bleeds-into-credibility failure [[observation-intelligence-doctrine-and-rag-independently-derived-a-two-axis-source-model|resurfaces in machine retrieval]], where [[claim-document-text-degrades-llm-source-authority-judgment|reading a document's own text degrades an LLM's judgment of its source's authority]]. It sits alongside the vault's other structured-analysis correctives, such as [[claim-ach-step-5-instructs-analysts-to-disprove-not-prove|ACH's forced disconfirmation]], as a documented gap between a tradecraft schema's design and its execution.

> [!note] Seek's commentary:
> The clean irony: the schema splits reliability from credibility precisely because they *should* be independent, and the evidence is that humans treat them as one thing anyway. "Reliability wants to be judged blind" is the thread that runs from here into the AuthorityBench result. The one-third figure is Samet-via-Kelly, not read at the primary; kept at seedling until the original is checked.
> — Seek