ChatGPT Reportedly Produced False Court Case Law Presented by Legal Counsel in Court

A lawyer in Mata v. Avianca, Inc. used ChatGPT for research. ChatGPT hallucinated court cases, which the lawyer then presented in court. The court determined the cases did not exist.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated false and misleading legal citations and court cases, which led the lawyer to submit a fraudulent brief to a federal court.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The lawyer blindly trusted and relied on ChatGPT's outputs for professional legal research without conducting secondary verification.
• 7.3 Lack of capability or robustness: The AI system failed to perform reliably, exhibiting severe hallucinations and errors by fabricating legal precedents.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by ChatGPT generating fabricated legal cases (hallucinations) and falsely confirming their authenticity, which was an unexpected outcome of using the system.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system involved is ChatGPT, a general-purpose chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to transparency obligations to ensure users are aware they are interacting with an AI.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Writing Assistant; Chatbot
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: Steven A. Schwartz, Peter LoDuca
• Alleged harmed parties: Steven A. Schwartz, Roberto Mata, Peter LoDuca

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: Yes, the AI system generated six completely fabricated court cases with fake quotes and citations, which were submitted as real evidence to a federal court.

Directly caused: ChatGPT directly generated fake legal precedents (e.g., Varghese v. China Southern Airlines) and falsely assured the lawyer of their authenticity.

Indirectly caused: The submission of fake cases polluted the court record, wasted the time of the judge and opposing counsel who searched for non-existent cases, and disrupted the legal proceedings.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 3

Potential causes

Management

• Lack of Firm AI Policies: The law firm had no internal guidelines or bans on using generative AI.
• Inadequate Supervision Protocols: Management failed to supervise the integration of AI tools in legal work.

Technology

• AI Hallucination: ChatGPT invented nonexistent legal cases and citations out of thin air.
• Deceptive Verification: The chatbot falsely insisted that the fabricated cases were real.
• Lack of Truth-Checking Design: Designed to sound impressive and predict text rather than ensure accuracy.

Data Inputs

• Unreliable Training Data: Ingested vast internet text without a mechanism to filter out errors.
• Lack of Database Integration: Not linked to verified databases like Westlaw or LexisNexis.

Human Factors

• Overreliance on AI Output: Attorney blindly trusted chatbot results without independent verification.
• Lack of AI Literacy: Attorney was unaware that generative AI could fabricate realistic text.
• Self-Verification Bias: Attorney asked the chatbot itself to verify if the cases were real.

Process and Methods

• Inadequate Research Verification: Firm process failed to mandate cross-checking citations in databases.
• Lack of Co-Counsel Review: Attorney of record signed and filed the brief without reviewing research.
• Improper Use of Chatbots: Using a conversational AI tool as a primary legal research engine.

Regulatory Environment

• Absence of Explicit AI Rules: Model Rules of Professional Conduct did not explicitly address AI use.
• Lack of Tech Competence Mandate: Vague ethical rules did not enforce strict proficiency in AI tools.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly consistent, detailed, and come from multiple reputable news organizations. They explicitly detail the AI system used (ChatGPT), the specific fake cases generated, the court proceedings, and the affidavits of the lawyers involved. There is no significant ambiguity regarding the facts of the incident.

A US lawyer used ChatGPT for legal research, leading to the submission of fabricated court cases to a federal court. While highlighting severe risks of AI hallucinations and professional overreliance, the incident represents a minor, localized disruption to judicial proceedings with negligible national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. Represents an ongoing professional reliability and training issue rather than an active national security crisis.
• Autonomy: Human-controlled. The AI system functioned as a writing assistant, with the human lawyer retaining final control over the submission of the document to the court.
• Novelty: First-of-its-kind. First widely documented and sanctioned instance of AI-generated legal hallucinations being submitted as genuine precedents in a federal court.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety domains were affected or threatened by this incident.
• Information security: Negligible. The incident involves a localized legal dispute without any involvement of state-sponsored disinformation campaigns, espionage, or intelligence compromises.
• Sovereignty: Minor. Briefly disrupted US federal court proceedings by introducing fabricated legal precedents, requiring judicial intervention and sanctions, but posed no threat to sovereign authority.
• Economic security: Negligible. No threats to national economic stability, strategic industries, or technological competitive advantage were identified.
• Societal stability: Negligible. The incident had no impact on societal stability, civil liberties, or population-scale human rights.