South African Court Found Self-Represented Litigant Relied on an AI-Hallucinated Judgment

Self-represented litigant Adriaan Johannes Roux relied on a nonexistent judgment in High Court submissions in Johannesburg, South Africa. The court attributed the false authority and unsound legal arguments to his admitted AI use and said addressing the material wasted court time and resources. The specific AI tools were not identified.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated discrete false information in the form of fabricated legal citations and a non-existent court judgment, which were then presented as factual in court.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The litigant uncritically trusted and relied on the AI's outputs without performing necessary verification, leading to severe legal and financial consequences.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the AI system generating unintended hallucinations (fabricated citations) during post-deployment use by a litigant.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The incident involves a Large Language Model used as a writing assistant and chatbot, which falls under generative AI systems subject to transparency obligations.

AI system and alleged parties

• AI system: unspecified AI
• AI purpose: Writing Assistant; Technical Text Generation
• Behaviour type: Assistant
• Alleged developer: Large language model developers, AI-assisted legal drafting system developers
• Alleged deployer: Pro se litigants, Adriaan Johannes Roux
• Alleged harmed parties: Parties to legal proceedings, Judicial integrity, High Court of South Africa, Gauteng Division, Johannesburg, High Court of South Africa, Epistemic integrity, Courts

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report explicitly describes financial losses in the form of punitive costs awarded against the applicant.

Directly caused: The applicant was ordered to pay the costs of the main application on the punitive attorney and client scale, as well as the ordinary costs of the counter-applications.

Inferred additional harm: The respondents incurred significant unrecoverable legal fees to defend against the baseless applications and address the fabricated submissions.

Epistemic

Reported: The report explicitly describes epistemic harm in the form of fabricated legal citations and a non-existent judgment presented to the court.

Directly caused: The AI system generated a completely fabricated judgment, 'Matshoba and Master of the High Court, Johannesburg and Others', with fake citations (2020 (1) SA 87 (GJ) and [2019] ZAGPJHC 186), which the litigant presented as authentic.

Indirectly caused: The court's time and resources were wasted verifying and addressing the non-existent authorities.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 3

Potential causes

Management

• Failure of Professional Duties: Negligent use of AI violated core duty to confirm authority accuracy.

Technology

• AI Hallucinations: LLMs generated highly convincing but entirely fabricated legal case citations.
• Predictive Statistical Output: LLMs generate responses based on statistical patterns, not legal reasoning.
• Lack of Source-Linked Outputs: General purpose AI lacks verified databases and audit trails for citations.

Data Inputs

• Unverified Training Data: AI model was trained on general data rather than a controlled legal database.

Human Factors

• Uncritical Reliance on AI: The practitioner accepted AI outputs without any human verification.
• Cognitive and Physical Stress: Litigant used AI to cope with hearing impairment and cognitive difficulties.
• Distrust of Legal Professionals: Litigant's belief that lawyers cannot be trusted led to self-use of AI.

Process and Methods

• Lack of Human Oversight: No process was in place to verify the accuracy of AI-generated submissions.
• Bypassing Procedural Remedies: Litigant used AI to draft urgent applications instead of standard procedures.
• Failure to Correct Errors: Doubling down on hallucinated cases instead of verifying court registries.

Regulatory Environment

• Insufficient AI Guidelines: Lack of specific court directives on the permissible use of generative AI.

Information quality

• Classification confidence: High
• Reason for confidence: The court judgment provides detailed, first-hand factual findings regarding the litigant's use of AI, the specific fabricated citations, and the resulting legal consequences. There is no ambiguity about the AI's role or the outcomes.

A South African litigant submitted AI-generated fabricated case citations to the High Court. The court identified the hallucinations and dismissed the case with punitive costs. The incident is a domestic civil matter managed through standard judicial procedures, presenting negligible national security implications.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: South Africa
• Alleged perpetrator: Adriaan [name withheld]

Threat characteristics

• Imminence: Long-term. The specific legal matter is resolved, representing no active crisis or immediate national security threat.
• Autonomy: Human-controlled. The AI acted as a writing assistant, and the human litigant made the final decision to submit the fabricated documents to the court.
• Novelty: Established threat. AI hallucinations in legal filings have occurred in multiple jurisdictions globally and represent an established issue of technology overreliance.

Impact by dimension

• Physical security: Negligible. No physical systems, weapons, or critical infrastructure were targeted or affected by this legal drafting incident.
• Information security: Negligible. The incident involves a single litigant using AI for private legal pleadings, with no indication of state-sponsored disinformation or intelligence compromise.
• Sovereignty: Negligible. While a court proceeding was disrupted by fabricated citations, the issue was easily managed within standard judicial procedures by dismissing the application.
• Economic security: Negligible. No strategic technology, critical supply chains, or financial systems were compromised or targeted in this incident.
• Societal stability: Negligible. This was an isolated civil legal dispute with no implications for mass surveillance, systemic discrimination, or civil unrest.