Study Highlights Persistent Hallucinations in Legal AI Systems

Stanford University’s Human-Centered AI Institute (HAI) conducted a study in which they designed a "pre-registered dataset of over 200 open-ended legal queries" to test AI products by LexisNexis (creator of Lexis+ AI) and Thomson Reuters (creator of Westlaw AI-Assisted Research and Ask Practical Law AI). The researchers found that these legal models hallucinate in 1 out of 6 (or more) benchmarking queries.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI systems fail to perform reliably under varying conditions, generating hallucinations and incorrect legal citations in a critical domain.

Additional risk subdomains

• 3.1 False or misleading information: The AI systems generate false legal precedents and fabricated citations, leading to inaccurate beliefs in users.
• 5.1 Overreliance and unsafe use: Lawyers and litigants place inappropriate trust in the AI outputs without verification, leading to legal sanctions and court errors.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The hallucinations and incorrect legal citations are unintended failures of the AI systems occurring after they have been deployed to users.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the systems as 'AI chatbots' and generative AI tools used for legal queries, which fall under the category of chatbots requiring transparency obligations.

AI system and alleged parties

• AI system: Ask Practical Law AI, GPT-4, Lexis+ AI, Westlaw AI-Assisted Research (LexisNexis, OpenAI, Thomson Reuters)
• AI purpose: Question Answering; Chatbot
• Behaviour type: Assistant
• Alleged developer: Thomson Reuters, LexisNexis
• Alleged deployer: Organizations requiring legal research, Legal professionals, Law firms
• Alleged harmed parties: Legal system, Legal professionals, Clients of lawyers

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the report notes that premium paid versions of AI tools perform better than free versions, risking digital and legal inequality.

Indirectly caused: The performance gap between paid and free AI models may lead to unequal legal outcomes for under-resourced litigants who cannot afford premium tools.

Civil rights

Reported: Yes, the reports mention risks to 'access to justice' for pro se and under-resourced litigants.

Indirectly caused: Litigants relying on faulty AI advice may have their legal rights compromised in court proceedings.

Epistemic

Reported: Yes, the reports explicitly describe AI tools fabricating legal cases, hallucinating citations, and providing incorrect legal advice.

Directly caused: AI models generated fictitious legal cases and misgrounded citations, misleading lawyers and litigants.

Indirectly caused: The proliferation of hallucinated legal precedents pollutes the legal information ecosystem and undermines trust in legal research.

Inferred additional harm: Widespread erosion of trust in legal documentation as AI-generated fabrications enter court records.

People affected

• Occurrences reported: 2
• People reportedly harmed: 2
• People reportedly exposed: 2

Potential causes

Management

• Opaque Performance Claims: Management markets tools as 'hallucination-free' without providing evidence.
• Neglecting Independent Evaluation: Firms adopt AI without developing hard metrics to measure actual accuracy.
• Inefficient Evaluation Methods: Firms spend excessive time testing tools without clear efficiency metrics.

Technology

• Inherent LLM Hallucinations: Generative AI models naturally tend to invent false legal information.
• Retrieval Mechanism Failures: RAG systems fail to retrieve the correct or best legal authorities.
• Model Sycophancy: AI models tend to agree with the user's incorrect assumptions.

Data Inputs

• Out-of-Date Training Data: Models lack recent legal developments, referencing replaced laws.
• Inapplicable Authority Retrieval: Systems retrieve semantically similar but legally irrelevant sources.
• US-Centric Training Bias: Chatbots default to US law without specifying jurisdiction differences.

Human Factors

• Overreliance on Well-Written Text: Users assume professional-looking AI outputs are correct and authoritative.
• Lack of User Legal Expertise: Pro se litigants cannot verify if AI-generated citations are real.
• Mistaken User Assumptions: Users input false premises which the AI naively validates.

Process and Methods

• Lack of Rigorous Benchmarking: Industry lacks public evaluations and hard metrics to assess reliability.
• Opaque AI Tool Design: Providers publish few details about model design and performance metrics.
• Inadequate Verification Processes: Users fail to verify every AI-generated proposition and citation.

Regulatory Environment

• Lagging Supervision Guidelines: Regulatory bodies only recently issued guidance on AI supervision duties.
• Lack of Standardized Benchmarks: No regulatory mandates exist for transparent reporting of AI error rates.
• Inconsistent Court Standing Orders: Judges independently issue varying disclosure rules for courtroom AI use.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed, citing specific academic studies (Stanford RegLab/HAI and Open University) and real-world legal cases (Manchester and New York). The findings are consistent across both reports.

A Stanford study revealed that legal AI tools frequently hallucinate, leading to fabricated citations being introduced into court proceedings. While representing a minor concern for judicial administration and access to justice, the national security implications are negligible to minor, with no threat to critical infrastructure, intelligence, or state sovereignty.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, United Kingdom
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing systemic challenge with AI reliability rather than an immediate national security crisis.
• Autonomy: Human-supervised. The AI systems act as research assistants requiring human verification and final decision-making, though users failed to supervise properly.
• Novelty: Evolved capability. Represents an evolution of known LLM hallucination issues into specialized, domain-specific retrieval-augmented generation (RAG) systems.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were impacted by the legal AI hallucinations.
• Information security: Negligible. The incident involves unintentional AI hallucinations in legal research, not state-sponsored information operations or intelligence compromise.
• Sovereignty: Minor. Minor disruption to judicial government functions as courts in the US and UK had to address fabricated legal precedents submitted by litigants.
• Economic security: Negligible. No significant threat to national economic stability or strategic technological competitive advantage.
• Societal stability: Minor. Potential minor risk to access to justice and equal legal outcomes for under-resourced litigants relying on free, less-reliable AI tools.