Michael Cohen, a former lawyer, used Google Bard to generate legal citations for a motion to end his supervised release. He mistakenly believed the AI was a search engine and provided the resulting fake citations to his attorney, David M. Schwartz, who included them in a court filing without verification. The presiding judge discovered the cases did not exist and requested an explanation, leading to public admission of the error and a subsequent ruling that declined to impose sanctions, citing a lack of bad faith.
Michael Cohen, former lawyer for Donald Trump, claims to have used Google Bard, an AI chatbot, to generate legal case citations. These false citations were unknowingly included in a court motion by Cohen's attorney, David M. Schwartz. The AI's misuse highlights emerging risks in legal technology, as AI-generated content increasingly infiltrates professional domains.
Risk classification
- Primary risk domain: 3 Misinformation
- Primary risk subdomain: 3.1 False or misleading information
The AI system generated fabricated legal citations that were presented as factual in a court filing, misleading the users and the court.
Additional risk subdomains
- 5.1 Overreliance and unsafe use: The lawyer and client overrelied on the AI system's outputs, submitting them to a federal court without verifying their authenticity.
Causal factors
- Entity: AI
- Intent: Unintentional
- Timing: Post-deployment
The risk arose from the AI system generating fabricated legal citations (hallucinations) during active use by an end-user.
EU AI Act risk tier
- Risk tier: 3 Limited Risk
Risk Level 3: Limited Risk. The system used is a general-purpose chatbot (Google Bard/ChatGPT), which falls under the category of chatbots and generative AI systems that require transparency but do not pose unacceptable or high risks by default.
AI system and alleged parties
- AI system: Google Bard (Google)
- AI purpose: Writing Assistant; Question Answering
- Behaviour type: Assistant
- Alleged developer: Google Bard, Google
- Alleged deployer: Michael Cohen, David M. Schwartz
- Alleged harmed parties: Michael Cohen, David M. Schwartz
Harm severity
Severity ratings are based on this scale.
Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.
- Physical: direct Negligible, indirect Negligible
- Infrastructure: direct Negligible, indirect Negligible
- Property: direct Negligible, indirect Negligible
- Financial: direct Negligible, indirect Negligible
- Environmental: direct Negligible, indirect Negligible
- Malicious content: direct Negligible, indirect Negligible
- Differential treatment: direct Negligible, indirect Negligible
- Civil rights: direct Negligible, indirect Negligible
- Democracy: direct Negligible, indirect Negligible
- Privacy: direct Negligible, indirect Negligible
- Psychological: direct Negligible, indirect Negligible
- Epistemic: direct Minor, indirect Negligible
- Child sexual exploitation and abuse: direct Negligible, indirect Negligible
Epistemic
Reported: Yes, the AI system fabricated non-existent legal cases and citations, which were then presented as factual in a federal court filing.
Directly caused: The AI generated three completely fabricated court decisions (including fake docket numbers and case summaries) which were submitted to the court.
People affected
- Occurrences reported: 1
- People reportedly harmed: 2
- People reportedly exposed: 2
Potential causes
Management
- Failure to Monitor Tech Risks: Legal team failed to establish policies regarding client-provided research.
Technology
- AI Hallucinations: Google Bard generated realistic but completely fictional court citations.
- High Error Rates in LLMs: LLMs produce false legal info between 69% and 88% of the time.
- Overconfident AI Outputs: Models fail to qualify answers with appropriate levels of confidence.
Data Inputs
- Sequence Prediction Flaws: Models guess text fragments based on training rather than retrieving facts.
Human Factors
- User Misunderstanding of AI: Cohen mistook the generative chatbot for a supercharged search engine.
- Lack of Legal Tech Awareness: Cohen was unaware of emerging trends and risks in generative AI tools.
- Overreliance and Blind Trust: Lawyer submitted client-provided AI citations without verifying them.
Process and Methods
- Lack of Human Verification: No process was used to verify the existence of the cited court cases.
- Poor Communication in Team: Lawyer falsely assumed citations came from co-counsel rather than client.
- Inadequate Quality Control: Law firm lacked standard procedures to check documents before filing.
Regulatory Environment
- Lack of AI Legal Standards: Courts are still grappling with how to regulate generative AI in filings.
Information quality
- Classification confidence: High
- Reason for confidence: The reports provide consistent, detailed, and unsealed court-documented accounts of the incident, including direct quotes from the judge, Michael Cohen, and his attorney. The facts of the AI's hallucination and its subsequent submission to the court are clear and undisputed.
A former lawyer and his attorney unintentionally submitted fabricated AI-generated legal citations to a U.S. federal court. While highlighting the risks of professional overreliance on generative AI and causing minor administrative disruption to judicial proceedings, the incident has negligible national security implications.
- Overall national security impact: Minor
- Scope: Single nation
- Primary target: United States
- Alleged perpetrator: Unknown
Threat characteristics
- Imminence: Long-term. Represents an ongoing strategic concern regarding AI overreliance in professional settings rather than an active, immediate national security crisis.
- Autonomy: Human-controlled. The AI functioned as an assistant to generate text under human direction, with humans retaining final control over the submission of the document.
- Novelty: Established threat. Similar incidents of lawyers submitting AI-generated fake citations had already occurred, making this an established issue of AI hallucination and user overreliance.
Impact by dimension
- Physical security: Negligible. No physical systems, critical infrastructure, or human safety were impacted or threatened by this legal filing error.
- Information security: Negligible. This was an unintentional user error involving a consumer AI tool, not an information warfare campaign, intelligence compromise, or state-sponsored disinformation.
- Sovereignty: Minor. The incident caused minor disruption to a federal court proceeding by introducing non-existent legal citations, wasting judicial time, but did not threaten core state authority.
- Economic security: Negligible. No financial system attacks, strategic tech theft, or loss of technological competitive advantage occurred.
- Societal stability: Negligible. No threats to social cohesion, systematic discrimination, mass surveillance, or civil unrest resulted from this isolated legal filing mistake.
.avif){kind=small}