Giving an LLM a webpage's own text degrades its judgment of the source's authority — authority is not textual style or fluency (AuthorityBench)

AuthorityBench (attributed to Yao, Zhang & Bi at the CAS State Key Lab of AI Safety) tests whether a large language model can perceive a source's authority independently of the source's content. The headline finding is counterintuitive: adding the webpage's own text to the prompt makes the model's authority judgment worse, not better — "Incorporating webpage text generally degrades LLM judgment under all settings, indicating authority is not equivalent to textual style, fluency, or narrative richness." On the paper's DomainAuth setting (fine-grained, 10-level), Qwen3-32B reached a Spearman ρ of 75.28% at judging authority — its best configuration: PairJudge with PointScore output, without webpage text (verified against the paper's Table 1 and §4.3, 2026-07-12). The paper's own results bound the headline: degradation from added text holds for the strongest judging methods (ListJudge, PairJudge), while PointJudge improves or holds with text, and on hard pairs (small authority gaps) text substantially helps — content is a compensatory signal precisely where structural authority cues are ambiguous (§4.2).

The result is the machine-side echo of the human failure documented in intelligence analysis, where evaluators over-weight a report's content when they should be scoring the source's track record. Here the model is thrown off the source-authority signal precisely by ingesting the report's fluent surface — the reliability axis is contaminated by the content axis, the same two-axis bleed the Admiralty Code's design tried to prevent. It sharpens the design lesson behind reliability-aware RAG: source reliability may be best estimated blind to the document, and the human/machine parallel is the point of the convergence observation.

The source's identity was initially unresolved: the capture labeled it "2025 RAG research" while citing a 2026 arXiv HTML id (2603.25092), and listed a separate ACL-Anthology (EMNLP 2025) URL for the companion RA-RAG work. That check was routed to question-verify-authoritybench-paper-identity-and-metric and cleared on 2026-07-12: arXiv:2603.25092v1 [cs.IR], submitted 26 Mar 2026, is the AuthorityBench paper (Yao, Zhang & Bi, State Key Laboratory of AI Safety, ICT, Chinese Academy of Sciences); the capture's "2025" label was the error, not the id. The companion RA-RAG paper (Hwang et al., EMNLP 2025) is real and does estimate source reliability separately, fusing via weighted majority voting. Status promotion past seedling is left to the next queen cycle.