Colorado Lawyer Filed a Motion Citing Hallucinated ChatGPT Cases

A Colorado Springs attorney, Zachariah Crabill, mistakenly used hallucinated ChatGPT-generated legal cases in court documents. The AI software provided false case citations, leading to the denial of a motion and legal repercussions for Crabill, highlighting risks in using AI for legal research.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated fabricated legal citations (false information) which the attorney relied upon, leading to legal and professional harms.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The attorney over-relied on ChatGPT, failing to verify its output before submitting it to the court.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was initiated by ChatGPT generating fabricated legal citations, which was an unexpected hallucination during a post-deployment search task.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system involved is ChatGPT, which is a general-purpose chatbot. According to the definitions, chatbots fall under Risk Level 3 (Limited Risk) and require specific transparency obligations.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Writing Assistant; Question Answering
• Behaviour type: Assistant
• Alleged developer: OpenAI, ChatGPT
• Alleged deployer: Zachariah Crabill
• Alleged harmed parties: Zachariah Crabill's client, Zachariah Crabill, Legal system

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report does not explicitly quantify the financial losses, but it describes indirect financial harm from the attorney losing his job and his client's motion being denied.

Indirectly caused: The attorney lost his job and was suspended, causing a loss of income. The client's motion regarding a car payment dispute was denied, which likely resulted in financial liability, though the exact amount is not specified.

Inferred additional harm: The client likely faced financial penalties or judgments from the denied motion, and the attorney suffered significant lost wages during his suspension.

Psychological

Reported: The report explicitly describes psychological distress experienced by the attorney.

Indirectly caused: Attorney Zachariah Crabill experienced significant stress, burnout, and 'horror' upon realizing the AI had fabricated citations, as well as the distress of losing his job and facing suspension.

Epistemic

Reported: Yes, the report explicitly describes epistemic harm in the form of fabricated legal citations submitted to a court of law.

Directly caused: ChatGPT generated several fictitious case citations, such as a fabricated 2014 insurance case, which did not exist in reality.

Indirectly caused: The fabricated citations were submitted to a Colorado court, polluting the legal record and requiring the judge and legal aides to spend time attempting to verify non-existent information.

People affected

• Occurrences reported: 1
• People reportedly harmed: 2
• People reportedly exposed: 2

Potential causes

Management

• Inadequate Supervision: The law firm failed to review the rookie attorney's work before submission.
• High Workload Pressure: Management assigned additional work to an already overwhelmed associate.

Technology

• AI Hallucination: ChatGPT generated realistic-looking but completely fabricated legal citations.
• Lack of Fact-Checking Mechanism: The AI tool did not verify the existence of the generated citations.

Data Inputs

• Lack of Real-Time Legal Data: The model was trained on general text rather than verified legal databases.

Human Factors

• Automation Bias and Trust: The attorney assumed the AI's output was correct due to prior accurate answers.
• Workplace Stress and Burnout: The attorney was overwhelmed by deadlines and a heavy workload.
• Lack of Research Experience: As a rookie lawyer, he lacked experience in drafting and verifying motions.

Process and Methods

• Failure to Verify Citations: The attorney did not check the cases in LexisNexis before filing.
• Lack of AI Usage Guidelines: No established procedures existed for using generative AI in legal work.

Regulatory Environment

• Absence of AI Court Rules: Courts lacked clear rules or disclosure mandates regarding generative AI use.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the incident from multiple sources, including direct quotes from the attorney involved, court documents, and the disciplinary ruling. The facts regarding the AI's role, the specific fabricated citations, and the resulting professional consequences are clear and uncontradicted.

A Colorado attorney's reliance on ChatGPT resulted in the submission of fabricated legal citations to a state court. While the incident highlights professional risks and reliability issues of generative AI, it has negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Zachariah Crabill

Threat characteristics

• Imminence: Long-term. The specific legal incident is resolved, but the broader issue of AI hallucinations in professional sectors remains a long-term reliability concern.
• Autonomy: Human-controlled. The AI system acted as a writing assistant, and the human user retained final control over whether to submit the generated text to the court.
• Novelty: Established threat. LLM hallucination is a well-documented technical limitation that has repeatedly led to factual errors in various user applications.

Impact by dimension

• Physical security: Negligible. The incident involves a local legal filing dispute with no physical security, critical infrastructure, or kinetic threat implications.
• Information security: Negligible. No state-sponsored disinformation, intelligence compromise, or classified data theft occurred. The incident was an isolated case of professional negligence using a public LLM.
• Sovereignty: Negligible. While the incident occurred within a state court system, it was a minor civil matter resolved through standard professional disciplinary procedures with no threat to state authority.
• Economic security: Negligible. The economic impact was limited to individual job loss and civil litigation penalties, posing no threat to strategic industries or national economic security.
• Societal stability: Negligible. The incident affected only the immediate parties to a civil lawsuit and did not threaten population-scale social cohesion, civil liberties, or human rights.