Five Reportedly Fictitious Citations Suspected of Being AI-Generated Led to a Costs Order in a London Housing Case

In a London housing case, England's High Court found that a barrister cited five reportedly fictitious cases and ordered her and the Haringey Law Centre, a legal aid charity, to pay £4,000 in wasted costs. Opposing counsel alleged AI use. The barrister denied using generative AI. The Divisional Court identified AI use as one of two possible explanations for the citations.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI systems generated highly plausible but completely false legal citations and fabricated quotes, which misled users and were subsequently presented as genuine evidence in court proceedings.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The lawyers and clients placed inappropriate trust and reliance on generative AI tools to conduct critical legal research without verifying the outputs against authoritative sources.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was initiated by generative AI systems hallucinating and outputting completely fictitious legal citations and quotes, which was an unexpected outcome of using the models for legal research.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The AI systems used were general-purpose generative AI chatbots and search assistants. According to the definitions, chatbots and general AI-generated content systems pose a moderate risk and fall under Risk Level 3 (Limited Risk).

AI system and alleged parties

• AI purpose: Writing Assistant; Technical Text Generation
• Behaviour type: Assistant
• Alleged developer: Large language model developers
• Alleged deployer: Sarah Forey, Legal professionals, Lawyers
• Alleged harmed parties: Parties to legal proceedings, London Borough of Haringey, Judicial integrity, High Court of Justice (England and Wales), Haringey Law Centre, Epistemic integrity, Courts

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report describes 2 occurrences causing a total reported financial loss of 11000 GBP, with an average loss of 5500 GBP per occurrence.

Indirectly caused: The submission of AI-generated fake cases resulted in a wasted costs order of 4000 GBP (split equally between the barrister and the law centre) and a 7000 GBP reduction in recovered costs in the Ayinde case, totaling 11000 GBP in indirect financial losses.

Inferred additional harm: The opposing parties in both cases likely incurred unquantified legal fees to research and disprove the dozens of fake citations.

Psychological

Reported: The report explicitly describes distress experienced by one of the lawyers involved in the incident.

Indirectly caused: Barrister Sarah Forey experienced significant distress, reporting that she was 'extremely upset' and 'blindsided' by the wasted costs proceedings and the threat of professional misconduct allegations.

Epistemic

Reported: The report explicitly describes epistemic harm where AI systems fabricated non-existent legal cases, citations, and quotes.

Directly caused: The AI models directly generated dozens of completely fictitious case citations and false legal propositions (e.g., 5 fake cases in Ayinde, 18 fake cases and multiple false quotes in Al-Haroun).

Indirectly caused: The fabrication of these cases polluted the legal proceedings, forcing opposing counsel and judicial assistants to spend significant effort verifying non-existent authorities.

People affected

• Occurrences reported: 2
• People reportedly harmed: 4
• People reportedly exposed: 10

Potential causes

Management

• Resource Constraints: Overstretched law centre had high caseloads and limited funding.
• Delegation to Lay Client: Solicitor relied on lay client to conduct and supply legal research.
• Poor Chambers Oversight: Chambers failed to supervise pupil barrister during busy practice.

Technology

• Generative AI Hallucinations: AI tools generated highly plausible but entirely fictitious case citations.
• Unreliable Search Summaries: Google or Safari AI summaries produced fake cases without user realizing.

Data Inputs

• Lack of Authoritative Data: Lawyers failed to cross-reference citations with authoritative sources.

Human Factors

• Overreliance on AI Output: Lawyers and clients trusted AI-generated citations without verification.
• Incompetence and Negligence: Junior or unqualified staff handled complex legal filings without guidance.
• Lack of Professional Diligence: Failure to check citations even after opposing counsel flagged them as fake.

Process and Methods

• Absence of Verification Step: No process existed to cross-check citations against official law databases.
• Inadequate Supervision: Pupil barrister written work was not checked by chambers supervisors.
• Deficient Correspondence Review: Firms dismissed opposing warnings as cosmetic errors without verifying.

Regulatory Environment

• Insufficient AI Guidance: Guidance promulgated by regulators was insufficient to prevent misuse.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide highly detailed, official court judgments and legal news articles detailing the exact nature of the fake citations, the parties involved, the financial penalties, and the admissions of AI use by the lay client and suspected use by the barrister.
• Ambiguities identified: The exact AI model used by the barrister in the Ayinde case is not definitively proven, though she admitted she may have used Google/Safari AI-generated summaries.
• Alternative interpretations: One could interpret the barrister's actions in the Ayinde case as purely human negligence without AI involvement, but the court found it highly probable that AI was the source of the highly specific, formulaic fake cases.

Two UK court cases involved lawyers submitting AI-generated fabricated legal citations. While this undermined the administration of justice locally, it was resolved through standard judicial and regulatory processes, posing negligible threat to broader national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding the integration of generative AI in legal and public administration processes.
• Autonomy: Human-controlled. The AI tools assisted with legal research, but humans maintained final control over whether to submit the generated text to the court.
• Novelty: Established threat. Similar incidents of AI-generated legal hallucinations being submitted to courts have occurred previously in other jurisdictions.

Impact by dimension

• Physical security: Negligible. No physical threat, kinetic attacks, or critical infrastructure disruption occurred in this incident.
• Information security: Negligible. This was an issue of professional negligence rather than a state-sponsored disinformation or intelligence compromise operation.
• Sovereignty: Minor. The submission of fabricated legal authorities affected the UK High Court, but the impact was minor and managed entirely through standard judicial and regulatory procedures.
• Economic security: Negligible. Financial impact was limited to minor legal costs and regulatory fines, with no threat to strategic industries or national economic stability.
• Societal stability: Negligible. The incident caused minor epistemic harm in specific court cases but did not threaten broader societal stability, civil liberties, or human rights.