A misinformation expert submitted an affidavit in a Minnesota legal case that included multiple non-existent academic citations. Attorneys for the plaintiffs argued these citations were AI-generated hallucinations from ChatGPT, which has cast doubt on the credibility of the expert's entire declaration and the legal proceedings.
In a legal case defending Minnesota’s deepfake election misinformation law, Stanford misinformation expert Professor Jeff Hancock's affidavit allegedly cited non-existent academic sources, potentially generated by ChatGPT. The reportedly fabricated citations appear to have undermined the credibility of his testimony.
Risk classification
- Primary risk domain: 3 Misinformation
- Primary risk subdomain: 3.1 False or misleading information
The AI system generated fabricated academic citations, which constitutes false and misleading information that was subsequently introduced into a federal court proceeding.
Additional risk subdomains
- 5.1 Overreliance and unsafe use: The expert relied on the AI-generated citations without verifying their authenticity, leading to severe reputational and legal consequences.
- 7.3 Lack of capability or robustness: The AI system exhibited a lack of robustness by hallucinating plausible-sounding but entirely fabricated academic sources.
Causal factors
- Entity: AI
- Intent: Unintentional
- Timing: Post-deployment
The generation of non-existent academic citations was an unintentional hallucination produced by the AI system during its post-deployment use.
EU AI Act risk tier
- Risk tier: 3 Limited Risk
Risk Level 3: Limited Risk. The system is a chatbot or general-purpose AI used to generate text, which falls under limited risk requiring transparency obligations.
AI system and alleged parties
- AI system: ChatGPT (OpenAI)
- AI purpose: Writing Assistant; Question Answering
- Behaviour type: Assistant
- Alleged developer: OpenAI, ChatGPT
- Alleged deployer: Jeff Hancock
- Alleged harmed parties: Mary Franson, Keith Ellison, Jeff Hancock, Christopher Kohls, Chad Larson
Harm severity
Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.
- Physical: direct Negligible, indirect Negligible
- Infrastructure: direct Negligible, indirect Negligible
- Property: direct Negligible, indirect Negligible
- Financial: direct Negligible, indirect Negligible
- Environmental: direct Negligible, indirect Negligible
- Malicious content: direct Negligible, indirect Negligible
- Differential treatment: direct Negligible, indirect Negligible
- Civil rights: direct Negligible, indirect Negligible
- Democracy: direct Negligible, indirect Negligible
- Privacy: direct Negligible, indirect Negligible
- Psychological: direct Negligible, indirect Negligible
- Epistemic: direct Minor, indirect Minor
- Child sexual exploitation and abuse: direct Negligible, indirect Negligible
Epistemic
Reported: The report explicitly describes epistemic harm through the fabrication of academic citations by an AI, which were then presented as true facts in a legal document.
Directly caused: The AI generated non-existent academic sources, such as 'The Influence of Deepfake Videos on Political Attitudes and Behavior', which were included in a formal legal declaration.
Indirectly caused: The inclusion of these fake citations misled the court and the public, casting doubt on the credibility of the expert's entire declaration and the research lab.
Inferred additional harm: N/A
People affected
- Occurrences reported: 1
- People reportedly harmed: 1
- People reportedly exposed: 3
Potential causes
Management
- Inadequate Delegation Oversight: Failure to supervise assistants or co-authors who may have used AI tools.
Technology
- LLM Citation Hallucination: ChatGPT generated plausible-sounding but entirely fictional academic citations.
Human Factors
- Failure to Verify References: The expert signed the declaration without verifying the cited academic papers.
- Overreliance on AI Tools: Assuming AI-generated text is accurate without performing manual fact-checks.
Process and Methods
- Inadequate Drafting Review: No formal process to cross-reference academic citations before legal submission.
Information quality
- Classification confidence: High
- Reason for confidence: The report clearly details the specific fake citations, the court case, the parties involved, and the consensus that they represent AI hallucinations. The facts are consistent across the reporting.
- Ambiguities identified: It is not 100% confirmed which specific LLM was used (though ChatGPT is suspected) or who exactly ran the prompt (Hancock or an assistant).
A prominent misinformation expert submitted a federal court declaration containing fabricated academic citations generated by ChatGPT. While highly embarrassing and damaging to the credibility of the legal proceedings regarding deepfake regulations, the incident represents a minor domestic legal disruption with negligible direct national security impact.
- Overall national security impact: Minor
- Response level: Moderate
- Scope: Single nation
- Primary target: United States
- Alleged perpetrator: Unknown
Threat characteristics
- Imminence: Long-term. The incident represents an ongoing concern regarding LLM reliability in professional settings rather than an active national security crisis.
- Autonomy: Human-controlled. The AI system was used as a writing assistant, and humans retained final control over the submission of the document.
- Novelty: Established threat. Similar incidents of AI-generated hallucinations being submitted in legal proceedings have occurred previously.
Impact by dimension
- Physical security: Negligible. No physical systems, kinetic threats, or critical infrastructure were affected by this incident.
- Information security: Negligible. The incident involves unintentional AI hallucinations in a domestic legal filing, not a state-sponsored information warfare campaign or intelligence compromise.
- Sovereignty: Minor. The submission of fabricated citations temporarily impacted the credibility of an expert witness in a federal court case concerning election deepfake laws.
- Economic security: Negligible. No strategic technologies, financial systems, or economic security interests were compromised.
- Societal stability: Negligible. The incident did not result in mass surveillance, societal instability, or systematic human rights violations.
.avif){kind=small}