Unrepresented Litigant Misled by ChatGPT-Generated False Legal Citations in Manchester Court

A litigant in person (LiP) in a Manchester civil case presented false legal citations generated by ChatGPT. It fabricated one case name and provided fictitious excerpts for three real cases, misleadingly supporting the LiP's argument. The judge, upon investigation, found the submissions to be inadvertent and did not penalize the LiP.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated fabricated legal citations and fictitious case excerpts, which constitutes the inadvertent generation of false and misleading information.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The litigant in person overrelying on ChatGPT for legal research and submitting its unverified, hallucinated outputs directly to a court of law.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by ChatGPT generating fabricated legal citations (hallucinations), which was an unexpected and unintended outcome of the system's operation after it was deployed.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system involved is ChatGPT, which is a chatbot. The EU AI Act classifies chatbots under Limited Risk, requiring specific transparency obligations to ensure users are aware they are interacting with an AI.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: Unnamed Manchester litigant
• Alleged harmed parties: Unnamed Manchester litigant, Manchester court system, General public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 3

Potential causes

Management

• Inadequate Risk Guidance: Judicial Office had not updated guidance to address AI risks for LiPs.

Technology

• AI Hallucination of Case Citations: ChatGPT fabricated case names and quotes that appeared legitimate.
• Plausible-Sounding AI Output: LLM generated natural-sounding but completely fictitious legal text.

Data Inputs

• Lack of Real-Time Legal Verification: AI model lacked active verification against official court databases.
• Training on Unfiltered Web Data: Model was trained to predict text patterns rather than verify facts.

Human Factors

• Unrepresented Litigant Naivety: Litigant in person trusted AI output without verifying legal citations.
• Overreliance on AI Chatbots: Litigant assumed the chatbot was a reliable legal search tool.

Process and Methods

• Lack of Case Citation Verification: No process was in place to verify LiP citations before court presentation.
• Inadequate Guidelines for LiPs: Existing handbooks did not warn against using AI-generated submissions.

Regulatory Environment

• Absence of AI Legal Regulations: No formal rules governing the use of AI tools in court submissions.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides a clear and consistent account of a specific court incident in Manchester. The role of ChatGPT in generating the fabricated citations is explicitly stated, and the outcome is clear. There are no conflicting accounts or major ambiguities regarding the core events.
• Alternative interpretations: The incident could be interpreted primarily as a human failure of overreliance and lack of verification by the litigant, rather than an AI safety failure, though the AI's generation of false information is the technical trigger.

An unrepresented litigant in a Manchester civil court inadvertently submitted fabricated legal citations generated by ChatGPT. The incident represents a minor, localized disruption to judicial administration with negligible wider national security implications, though it highlights long-term challenges regarding the verification of AI-generated content in government and legal processes.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: Litigant in person

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic concern regarding the integration of unverified AI tools in legal and public administrative processes.
• Autonomy: Human-controlled. The AI acted as an assistant generating text, while the human litigant made the final decision to submit the output to the court.
• Novelty: Evolved capability. Represents an evolved threat where generative AI hallucinations are introduced into formal state and judicial processes, establishing new challenges for court administration.

Impact by dimension

• Physical security: Negligible. No physical systems, infrastructure, or human safety elements were affected or threatened by this incident.
• Information security: Negligible. This was an isolated, inadvertent use of AI by an individual in a civil case, not a coordinated disinformation campaign or intelligence compromise.
• Sovereignty: Minor. The integrity of a local civil court proceeding was minorly disrupted by the submission of fabricated legal citations, though resolved quickly without systemic impact.
• Economic security: Negligible. The incident has no implications for strategic industries, economic stability, or technological competitive advantage.
• Societal stability: Negligible. There was no threat to societal stability, civil liberties, or mass public safety.