Westlaw CoCounsel Reportedly Generated False Legal Quotations in United States v. Farris

In United States v. Farris, court-appointed attorney Steven N. Howe admitted using Westlaw CoCounsel to draft appellate briefs that reportedly included AI-generated false quotations and misleading descriptions of precedent. The filing disrupted Farris's appeal, leading the court to appoint new counsel and restart briefing, while Howe lost his compensation and faced possible disciplinary action.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated false legal citations and fabricated direct quotations, which constitutes discrete false or misleading information that misled the attorney and was submitted to the court.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The attorney reflexively relied on the AI-generated draft without verifying the accuracy of the citations, failing to meet his ethical obligations of oversight.
• 7.3 Lack of capability or robustness: The AI system failed to perform reliably by hallucinating non-existent quotations and misrepresenting actual case holdings.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was initiated by the AI system generating hallucinated legal citations and false quotations, which was an unintentional outcome of its text generation process.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The AI system is used as a text generator and legal assistant, which falls under Limited Risk (Level 3) as it generates content and requires transparency regarding its AI nature.

AI system and alleged parties

• AI system: Westlaw CoCounsel (Thomson Reuters)
• AI purpose: Writing Assistant; Technical Text Generation
• Behaviour type: Assistant
• Alleged developer: Thomson Reuters, Large language model developers, AI research tool developers
• Alleged deployer: Steven N. Howe, Legal professionals, Law firms
• Alleged harmed parties: United States Court of Appeals for the Sixth Circuit, Legal clients, Judicial integrity, John C. Farris, Epistemic integrity

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly harmed: 2
• People reportedly exposed: 2

Potential causes

Management

• Lack of firm AI policies: The law office lacked policies governing the use and validation of AI tools.

Technology

• AI hallucination of quotes: CoCounsel generated fake direct quotes and mischaracterized legal holdings.
• Misleading citation generation: The tool associated real citations with completely fabricated text.

Data Inputs

• Raw court document uploads: Staff uploaded district court documents directly without pre-filtering.

Human Factors

• Overreliance on AI tool: Attorney trusted the AI output without performing necessary verification.
• Lack of AI familiarity: Attorney used the tool for the first time without understanding its limits.
• Inadequate staff supervision: Relying on non-attorney staff to run AI queries without direct oversight.

Process and Methods

• Lack of verification process: No systematic process was in place to verify AI-generated legal citations.
• Flawed drafting process: Relying on AI for initial drafts without subsequent manual quote checking.

Regulatory Environment

• Lack of clear AI guidelines: Absence of strict court rules or specific guidelines on AI use in briefs.

Information quality

• Classification confidence: High
• Reason for confidence: The report is a formal judicial opinion from the Sixth Circuit Court of Appeals, providing clear, verified facts about the attorney's actions, the specific AI-generated errors, and the resulting consequences. There is no conflicting information, and the attorney admitted to the facts.

An attorney submitted appellate briefs containing AI-generated false legal citations to a federal court. While this disrupted judicial resources and delayed a criminal appeal, it was managed through standard court procedures and carries negligible national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. The specific incident has been resolved by the court, representing a long-term strategic need for legal system guidelines rather than an active security crisis.
• Autonomy: Human-supervised. The AI system autonomously generated the text and citations, which the human attorney was responsible for supervising and verifying before submission.
• Novelty: Established threat. Generative AI hallucinations in legal filings are a well-documented issue with multiple prior cases of attorneys submitting fabricated citations.

Impact by dimension

• Physical security: Negligible. The incident involves legal brief preparation and has no connection to physical systems, kinetic threats, or critical infrastructure.
• Information security: Negligible. No evidence of coordinated disinformation, foreign influence operations, or compromise of classified intelligence systems.
• Sovereignty: Minor. The submission of AI-generated false legal citations mildly disrupted federal judicial operations, requiring the Sixth Circuit Court of Appeals to appoint new counsel and delay the appeal, which was managed through standard court procedures.
• Economic security: Negligible. No impact on national economic stability, strategic industries, or international technological competitive advantage.
• Societal stability: Negligible. While an individual defendant's appeal was delayed, this isolated legal error does not represent a large-scale threat to societal stability or systemic human rights.