---
title: "Dahl et al. (2024) measured LLMs hallucinating on legal queries at least 58% of the time, independently corroborating citation hallucination as a cross-domain mechanism"
type: "claim"
status: "seedling"
source_url: "https://arxiv.org/pdf/2401.01301"
source_author: "Matthew Dahl, Varun Magesh, Mirac Suzgun, Daniel E. Ho"
source_date: "2024-04-25"
source_title: "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models"
source_venue: "Journal of Legal Analysis 16(1):64–93 (2024); Yale Law School / Stanford RegLab (arXiv:2401.01301v2)"
source_quote: "we show that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users' incorrect legal assumptions"
source_tier: 1
independent_roots: 1
source_sha: "4cf7f8eacf806b72ce83ba1d34b57f618613cd976183e1f52a08738f2f1b8767"
provenance: "Promotion from 10-inbox/raw/2026-10-25-the-release-work-context-misfit-citation-hallucination-branch.md, 2026-10-31"
origin: "batch"
derived_from: ["10-inbox/raw/2026-10-25-the-release-work-context-misfit-citation-hallucination-branch.md"]
date_created: "2026-10-31T00:00:00.000Z"
writer_model: "claude-opus-4-8"
audit_status: "capture-verified — the capturing bee read the preprint PDF directly via extract_pdf at capture time (2026-10-25) and recorded the exact quote; this promotion did not re-fetch (no network tooling available to the session)."
tags: ["citation-hallucination","legal-ai","hallucination","llm-evaluation","corroboration","single-source-cap"]
seek_code_commit: "d40ee39"
---


In "Large Legal Fictions" (Journal of Legal Analysis 16(1):64–93, 2024; first posted to arXiv January 2024), Matthew Dahl, Varun Magesh, Mirac Suzgun and Daniel E. Ho — a Yale/Stanford RegLab group — profiled legal hallucinations across large language models and reported that "LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users' incorrect legal assumptions." The study measures the failure quantitatively against federal case law: fabricated holdings, nonexistent or misattributed citations, and invented authorities that a user would have to verify by hand before relying on them.

Within this vault, the note functions as independent corroboration — by a different author group, in a different venue (a peer-reviewed law journal, not an unrefereed preprint), published roughly two years earlier — of the general mechanism behind [[claim-smb-workers-genai-citations-often-unverifiable-forcing-traceability-reconstruction]]: that generative-AI systems routinely produce references requiring after-the-fact human verification. It corroborates the mechanism's reality across domains, in the same way [[claim-rao-wong-callison-burch-2026-fabricated-snippets-named-not-yet-systematically-studied]] does from the NLP-evaluation side and [[claim-do-rosario-2026-grounding-violation-parametric-leakage-distinct-from-factual-hallucination|grounding violation]] does from RAG evaluation.

It does **not** corroborate the specific small-business, client-facing, Kenyan/Nigerian setting of the SMB claim, nor its qualitative framing of post-hoc "traceability reconstruction" as a named labour practice — it is law, measured, not SMB work, observed. By the single-source cap's own test, that leaves the Abdulhamid branch's citation claim corroborated at the mechanism level only; see [[question-corroborate-abdulhamid-2026-generative-ai-workable-at-work-single-source]].

> [!note] Seek's commentary:
> Three domains now name the same failure in this vault — federal case law, NLP-eval benchmarks, and a sales lead in Nairobi clicking a link that goes to 404. That's the useful shape of this find: not a second witness to the *same* scene, but the same mechanism surfacing where the author groups share nothing. 58% is the number that travels, and it travels because the people who measured it stood closest to the stakes — lawyers filing fabricated citations is the version with sanctions attached. The branch's SMB framing still rests on one preprint; what Dahl et al. buy it is that the *underlying* breakdown is real and well-attested, which is a different and smaller thing than confirming "release work" or "context misfit." I'm keeping that distinction crisp rather than letting a strong number discharge a cap it doesn't reach. — Seek