A law firm, Boies Schiller Flexner, submitted a legal brief containing fabricated case citations and errors, which the partner responsible attributed to the improper use of AI during the drafting process. The court denied the firm's request to replace the brief and considered monetary sanctions, marking a significant instance of AI-related error in legal practice.
Boies Schiller Flexner filed a respondents' brief in Bixler v. Church of Scientology International that was reportedly prepared with AI assistance and allegedly contained material citation errors. Opposing counsel reportedly identified mischaracterized and mistitled authorities, including one nonexistent case. Partner John Kucera accepted responsibility for failing to verify the citations and sought to replace the brief; the court denied that request and later considered monetary sanctions.
Risk classification
- Primary risk domain: 3 Misinformation
- Primary risk subdomain: 3.1 False or misleading information
The AI system inadvertently generated fabricated legal citations and false case holdings, which were then submitted to a court of law as factual.
Additional risk subdomains
- 5.1 Overreliance and unsafe use: The lawyer overrelied on the AI system to draft the brief and failed to scrupulously proofread or verify the citations before filing.
Causal factors
- Entity: AI
- Intent: Unintentional
- Timing: Post-deployment
The incident was caused by an AI system generating hallucinated and fabricated legal citations during the drafting of a brief, which was an unexpected and unintentional outcome of its use post-deployment.
EU AI Act risk tier
- Risk tier: 4 Minimal or No Risk
Minimal or No Risk: The AI system used was a writing or drafting assistant, which generally poses minimal risk and has no additional regulatory obligations under the Act.
AI system and alleged parties
- AI system: unspecified
- AI purpose: Writing Assistant; Technical Text Generation
- Behaviour type: Assistant
- Alleged developer: Generative AI developers
- Alleged deployer: Lawyers, Law firms, John Kucera, Boies Schiller Flexner LLP
- Alleged harmed parties: Plaintiffs in Bixler v. Church of Scientology International, Judicial integrity, Epistemic integrity, Clients of lawyers, Church of Scientology International, Church of Scientology Celebrity Centre International, California Court of Appeal, Second Appellate District
Harm severity
Severity ratings are based on this scale.
Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.
- Physical: direct Negligible, indirect Negligible
- Infrastructure: direct Negligible, indirect Negligible
- Property: direct Negligible, indirect Negligible
- Financial: direct Negligible, indirect Negligible
- Environmental: direct Negligible, indirect Negligible
- Malicious content: direct Negligible, indirect Negligible
- Differential treatment: direct Negligible, indirect Negligible
- Civil rights: direct Negligible, indirect Negligible
- Democracy: direct Negligible, indirect Negligible
- Privacy: direct Negligible, indirect Negligible
- Psychological: direct Negligible, indirect Negligible
- Epistemic: direct Minor, indirect Negligible
- Child sexual exploitation and abuse: direct Negligible, indirect Negligible
Epistemic
Reported: Yes, the report explicitly describes the fabrication of legal citations and case holdings in a court filing.
Directly caused: The AI generated completely fabricated case citations and mistitled existing cases, which were then submitted to a court of law.
People affected
- Occurrences reported: 1
- People reportedly harmed: 4
- People reportedly exposed: 10
Potential causes
Management
- Ineffective Oversight: Responsible partner failed to properly oversee the brief preparation process.
- Inadequate Policy Enforcement: Existing AI policies and trainings were not successfully enforced.
Technology
- AI Hallucinations: AI generated a completely made-up case and incorrect citations.
- Flawed AI Text Generation: AI mistitled cases and mischaracterized legal holdings.
Human Factors
- Lack of Personal Proofreading: The partner failed to scrupulously proofread and cite check the brief.
- Overreliance on AI Outputs: Lawyers trusted the AI-generated brief without verifying the citations.
Process and Methods
- Inadequate Citation Verification: The process for verifying citations failed to catch the AI-generated errors.
Information quality
- Classification confidence: High
- Reason for confidence: The report clearly outlines the incident where a lawyer submitted a brief with fabricated citations generated by AI. The roles of the AI, the lawyer, and the opposing party are well-defined, leaving little ambiguity about the nature of the failure.
- Ambiguities identified: The specific AI tool used was not identified in the report.
A US law firm partner submitted a legal brief containing fabricated AI-generated citations in a civil lawsuit. While representing a significant professional error and highlighting risks of overreliance on AI in the legal sector, the incident has negligible national security implications.
- Overall national security impact: Negligible
- Scope: Single nation
- Primary target: United States
- Alleged perpetrator: Individual
Threat characteristics
- Imminence: Long-term. Represents an ongoing professional capability and standards concern rather than an active crisis requiring urgent national response.
- Autonomy: Human-controlled. The AI was used as a writing assistant to generate text, requiring human review and filing.
- Novelty: Established threat. AI-generated hallucinations in legal filings have occurred in several high-profile cases previously.
Impact by dimension
- Physical security: Negligible. The incident is a legal drafting error in civil litigation with no impact on physical security or critical infrastructure.
- Information security: Negligible. No intelligence compromise, classified data theft, or state-sponsored information warfare operations were involved.
- Sovereignty: Negligible. While it occurred within a court of law, it was an isolated professional error in civil litigation and did not disrupt or compromise core government operations.
- Economic security: Negligible. No strategic technology theft, critical supply chain vulnerabilities, or broader economic stability threats occurred.
- Societal stability: Negligible. The incident did not threaten social cohesion, civil liberties, or population safety.
.avif){kind=small}