Crown Prosecution Service Reportedly Filed Purportedly AI-Generated False Citations in Two Extradition Appeals

In joined appeals involving Andreea-Maria Tobosaru and Marian Tofan, the Crown Prosecution Service reportedly filed grounds of opposition containing two nonexistent authorities that likely originated from purported generative AI. The citations also appeared in a later document. The High Court said they did not affect the appeals, and the CPS apologized, citing inadequate verification by a reviewing lawyer.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system inadvertently generated incorrect and fabricated legal citations, which were then included in formal court submissions.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The CPS lawyers relied on the AI-generated document without verifying its accuracy before filing it in court.
• 7.3 Lack of capability or robustness: The AI system failed to perform reliably by hallucinating non-existent legal authorities during legal research.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by an AI system generating non-existent legal citations (hallucinations) during post-deployment use.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a generative AI used to compose text, which falls under AI-generated content obligations requiring transparency.

AI system and alleged parties

• AI system: generative artificial intelligence
• AI purpose: Writing Assistant; Technical Text Generation
• Behaviour type: Assistant
• Alleged developer: Large language model developers
• Alleged deployer: Lawyers, Crown Prosecution Service
• Alleged harmed parties: Marian Tofan, Judicial integrity, High Court of Justice (England and Wales), Extradition appellants, Epistemic integrity, Courts, Andreea-Maria Tobosaru

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: Yes, the report describes epistemic harm in the form of fabricated legal citations and non-existent authorities generated by the AI.

Directly caused: The AI system generated non-existent legal authorities and citations that were included in formal court submissions prepared by the CPS.

Indirectly caused: The fabricated citations were subsequently used in a further document in the appeal prepared by junior counsel.

People affected

• Occurrences reported: 1
• People reportedly exposed: 2

Potential causes

Management

• Inadequate Risk Assessment: CPS lacked proper risk assessment for using generative AI in court work.

Technology

• Generative AI Hallucination: The AI generated non-existent legal citations and authorities.

Human Factors

• Lack of Verification: The reviewing lawyer failed to check the accuracy of the document.
• Inadequate Checking by Counsel: Junior counsel did not verify authorities before preparing documents.

Process and Methods

• Flawed Document Review Process: The CPS process did not catch AI errors before filing and serving.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, direct facts about the incident, including the organization involved (CPS), the nature of the AI error (hallucinated citations), how it was discovered, and the lack of actual harm. There is no conflicting information or major ambiguity regarding what transpired.
• Ambiguities identified: The specific AI model or platform used by the CPS is not named.
• Alternative interpretations: None. The facts of the case are straightforward and accepted by all parties involved.

The UK Crown Prosecution Service used generative AI to draft extradition documents containing fabricated legal citations. While this compromised the quality of submissions to a core government function, it was caught by human review and did not result in systemic harm, representing a minor national security impact managed through standard legal procedures.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic concern regarding the integration and oversight of AI tools in government workflows rather than an active crisis.
• Autonomy: Human-controlled. The AI tool was used as a writing assistant to generate drafts, with humans retaining the responsibility to review, verify, and submit the final documents.
• Novelty: Established threat. AI hallucinations and the generation of fabricated legal citations are well-documented phenomena that have occurred in multiple legal contexts previously.

Impact by dimension

• Physical security: Negligible. The incident involved generative AI draft court documents and did not impact physical systems, critical infrastructure, or human safety.
• Information security: Negligible. No information warfare, state-sponsored disinformation, or compromise of classified intelligence capabilities occurred.
• Sovereignty: Minor. The incident impacted the Crown Prosecution Service, a core government agency, by introducing fabricated legal citations into extradition court documents. However, the error was caught and corrected with no disruption to sovereign authority.
• Economic security: Negligible. There was no threat to economic stability, financial systems, strategic industries, or technological competitive advantage.
• Societal stability: Negligible. While the incident occurred within the justice system, the errors were identified before causing systemic injustice, and there was no mass human rights violation or societal instability.