An assistant US attorney faces potential sanctions after filing a court brief containing fabricated legal citations and quotations, suspected to have been generated by an AI tool.
In Fivehouse v. U.S. Department of Defense, Assistant U.S. Attorney Rudy Renfer filed a response brief that the court reportedly said contained fabricated quotations and misstatements of case holdings. Renfer reportedly admitted he had used AI to help draft the inaccurate filing after losing an earlier draft. The court reportedly also identified similar fabricated quotations in other filings signed by Renfer.
Risk classification
- Primary risk domain: 3 Misinformation
- Primary risk subdomain: 3.1 False or misleading information
The AI system generated fabricated legal citations and quotations in a court brief, leading to false representations being submitted to a federal court.
Additional risk subdomains
- 5.1 Overreliance and unsafe use: The attorney over-relied on the AI tool without verifying the citations, leading to court sanctions.
Causal factors
- Entity: AI
- Intent: Unintentional
- Timing: Post-deployment
The generation of fabricated legal citations is an unintentional failure of a deployed AI writing assistant tool.
EU AI Act risk tier
- Risk tier: 3 Limited Risk
Limited Risk: The suspected system is a text generator producing AI-generated content, which falls under Limited Risk (Level 3) due to transparency obligations regarding AI-generated text.
AI system and alleged parties
- AI purpose: Writing Assistant; Technical Text Generation
- Behaviour type: Assistant
- Alleged developer: Unknown large language model developers
- Alleged deployer: United States Attorney's Office for the Eastern District of North Carolina, Rudy Renfer
- Alleged harmed parties: Judicial integrity, Epistemic integrity, Derence V. Fivehouse
Harm severity
Severity ratings are based on this scale.
Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.
- Physical: direct Negligible, indirect Negligible
- Infrastructure: direct Negligible, indirect Negligible
- Property: direct Negligible, indirect Negligible
- Financial: direct Negligible, indirect Negligible
- Environmental: direct Negligible, indirect Negligible
- Malicious content: direct Negligible, indirect Negligible
- Differential treatment: direct Negligible, indirect Negligible
- Civil rights: direct Negligible, indirect Negligible
- Democracy: direct Negligible, indirect Negligible
- Privacy: direct Negligible, indirect Negligible
- Psychological: direct Negligible, indirect Negligible
- Epistemic: direct Minor, indirect Minor
- Child sexual exploitation and abuse: direct Negligible, indirect Negligible
Epistemic
Reported: The report explicitly describes the submission of fabricated quotations and misstatements of case holdings to a federal court.
Directly caused: The assistant US attorney submitted a brief containing multiple fabricated quotes from circuit court opinions and the Code of Federal Regulations.
Indirectly caused: The court's record was contaminated with false legal precedents, requiring a show-cause hearing to address the fabrications.
Inferred additional harm: The integrity of the judicial process was temporarily compromised, and court resources were wasted verifying and correcting the fabricated citations.
People affected
- Occurrences reported: 1
- People reportedly harmed: 1
- People reportedly exposed: 2
Potential causes
Management
- Lack of Office Oversight: Leadership failed to monitor and ensure quality control of attorney filings.
Technology
- Suspected AI Tool Usage: Unverified use of AI tools may have generated fabricated legal citations.
Data Inputs
- Unverified External Sources: Incorporating text from unchecked external sources or unfinalized drafts.
Human Factors
- Inadvertent Draft Filing: Attorney signed and filed an unfinalized draft containing errors.
- Inadequate Verification: Attorney failed to verify quotes and holdings before submitting to court.
Process and Methods
- Insufficient Review Process: The office lacked a robust process to peer-review filings before submission.
Information quality
- Classification confidence: Medium
- Reason for confidence: While the reports detail the fabricated citations and the court's response, the use of an AI tool is inferred based on the nature of the fabrications (hallucinations) and the context of similar legal AI incidents. The US Attorney's office did not confirm or deny the use of AI.
- Ambiguities identified: Whether an AI tool was actually used, and if so, which specific model or platform was employed.
- Alternative interpretations: The attorney could have manually fabricated the quotes or copied them from another human source, as suggested by the plaintiff's statement about the 'provenance of the fabricated language'.
- Missing information: Confirmation of whether an AI tool was used, and the specific name of the AI model or platform.
An assistant US attorney filed a federal court brief containing fabricated citations likely generated by an AI tool. While this represents a minor disruption to judicial operations and highlights risks of AI overreliance within government functions, it does not pose a significant threat to national security.
- Overall national security impact: Minor
- Scope: Single nation
- Primary target: United States
- Alleged perpetrator: Individual
Threat characteristics
- Imminence: Long-term. Represents an ongoing systemic issue of AI overreliance in legal drafting rather than an active national security crisis.
- Autonomy: Human-supervised. AI acted autonomously to generate text, which was then filed by a human supervisor without proper verification.
- Novelty: Established threat. Similar incidents of AI-generated legal hallucinations have occurred previously in other court cases.
Impact by dimension
- Physical security: Negligible. No physical security or critical infrastructure threats identified in the incident.
- Information security: Negligible. No intelligence compromise, classified data theft, or strategic information warfare operations indicated.
- Sovereignty: Minor. Minor disruption to federal court proceedings and US Attorney's Office operations, managed through standard judicial show-cause procedures.
- Economic security: Negligible. No strategic technology theft, financial system attacks, or economic security implications.
- Societal stability: Negligible. No threats to societal stability, civil liberties, or large-scale population safety.
.avif){kind=small}