Factum in Ko v. Li Allegedly Contains AI-Generated Case Law Citations

In Ko v. Li, 2025 ONSC 2766, the Ontario Superior Court of Justice identified multiple inaccurate or nonexistent case citations in a factum submitted by counsel for the applicant. Justice FL Myers questioned whether generative AI was used in drafting the document, noting the cited cases were either irrelevant, misrepresented, or did not exist. The court ordered the lawyer to show cause why she should not be cited for contempt, citing duties of accuracy and technological competence.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI systems generated false and misleading legal citations (hallucinations) that were presented as factual precedents in court filings.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The lawyers over-relied on the AI tools to conduct legal research and generate briefs without verifying the accuracy of the citations.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by AI systems generating fabricated legal citations, which was an unintentional and unexpected outcome of their deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The AI systems used, such as Google Gemini and ChatGPT, are chatbots and content generation tools that pose a moderate risk and require transparency, particularly regarding their propensity to generate hallucinated content.

AI system and alleged parties

• AI system: generative artificial intelligence
• AI purpose: Writing Assistant; Content Search
• Behaviour type: Assistant
• Alleged developer: Unspecified large language mode developer
• Alleged deployer: Jisuh Lee
• Alleged harmed parties: Ontario Superior Court of Justice, Legal integrity, Jisuh Lee, Hanna Ko

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the US court ordered the law firms to jointly pay $31,100 in the defendant's legal fees.

Directly caused: The law firms were ordered to pay $31,100 in legal fees due to the submission of the fake brief. Average financial loss per occurrence is $15,550.

Inferred additional harm: The clients and law firms likely incurred additional unquantified financial losses due to delayed proceedings, lost discovery relief, and time spent addressing the hallucinations.

Epistemic

Reported: Yes, the AI systems fabricated non-existent legal cases and citations (hallucinations), which were then presented as real precedents in court briefs.

Directly caused: The AI tools generated multiple completely fabricated legal citations (e.g., at least two non-existent cases in the California brief, and multiple fake cases/hyperlinks in the Toronto factum).

Inferred additional harm: The propagation of fake legal precedents could pollute legal databases if not caught, leading to further reliance on false information by other legal researchers.

People affected

• Occurrences reported: 2
• People reportedly harmed: 5
• People reportedly exposed: 15

Potential causes

Management

• Inadequate Staff Supervision: Partners failed to supervise associates or clerks using generative AI tools.
• Failure to Educate Staff on AI Risks: Firms did not train lawyers on the risks of generative AI hallucinations.

Technology

• Generative AI Hallucinations: AI tools fabricated non-existent legal citations and hyperlinked them.
• Inaccurate Search Results: AI tools mapped real case names to unrelated or opposite legal propositions.
• Lack of AI Origin Tracking: AI tools did not clearly flag or watermark generated text as synthetic.

Data Inputs

• Unverified Hyperlinks: AI generated plausible-looking hyperlinks that led to 404 errors or wrong pages.
• Incomplete Legal Databases: AI models trained on incomplete datasets hallucinated missing legal precedents.
• Flawed Reference Mapping: AI mapped citations to incorrect external URLs or unrelated database entries.

Human Factors

• Overreliance on AI Outputs: Lawyers assumed AI-generated research was accurate without manual verification.
• Failure to Verify Citations: Attorneys submitted briefs without reading the cited cases to ensure existence.
• Failure to Disclose AI Use: Attorney shared AI outlines with colleagues without disclosing the AI source.

Process and Methods

• Lack of Human-in-the-Loop Review: No process was in place to verify the validity of AI-generated legal research.
• Inadequate Quality Control: Corrected briefs were submitted with remaining AI errors due to poor review.
• Absence of AI Usage Guidelines: Law firms lacked standardized procedures for integrating AI into legal drafting.

Regulatory Environment

• Ineffective Professional Oversight: Bar association rules failed to deter lawyers from submitting fake cases.
• Lack of AI Auditing Standards: No formal regulatory standards exist for validating AI legal research tools.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, detailed accounts of the two legal cases, including specific court citations, names of law firms, the specific AI tools used, and the exact nature of the hallucinations and subsequent judicial sanctions.
• Ambiguities identified: The exact AI model used in the Toronto case is not definitively confirmed, though the judge strongly suspects ChatGPT or a similar generative AI tool.

Lawyers in the US and Canada submitted court filings containing fabricated legal citations generated by AI tools, resulting in judicial sanctions. This incident represents a minor national security concern, specifically affecting sovereignty and government functions by temporarily disrupting court proceedings, but was easily managed through standard judicial oversight and disciplinary mechanisms.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: United States
• Other affected: Canada
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing professional standards and technological literacy concern rather than an active national security crisis.
• Autonomy: Human-controlled. The AI tools acted as drafting assistants, and human lawyers retained final authority and responsibility for submitting the court filings.
• Novelty: Established threat. AI hallucinations in court filings are an established threat with several well-documented prior occurrences in the legal industry.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic safety elements were threatened or impacted by the incident.
• Information security: Negligible. No state-sponsored disinformation campaigns, intelligence compromises, or classified model thefts were associated with this incident.
• Sovereignty: Minor. The administration of justice was temporarily disrupted in domestic courts due to fabricated citations, but the impact was minor and managed within standard judicial procedures.
• Economic security: Negligible. The economic impact was limited to minor individual legal sanctions and fees, with no threat to strategic industries or national economic security.
• Societal stability: Negligible. There was no threat to societal stability, civil liberties, or population-scale human rights.