Judges in New Jersey and Mississippi Admit AI Tools Produced Erroneous Federal Court Filings

Two U.S. federal judges, Julien Neals (D.N.J.) and Henry Wingate (S.D. Miss.), reportedly admitted that staff in their chambers used AI tools, ChatGPT and Perplexity, to draft rulings containing purportedly fabricated quotes, cases, and parties. The erroneous filings were reportedly docketed before review, prompting Senate and judicial inquiries. Both rulings were retracted.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI systems inadvertently generated fabricated legal citations, quotes, and case details, which were then published in official court dockets, spreading false information within the judicial system.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Court staff overrelied on generative AI tools for legal research and drafting without verifying the outputs, leading to the docketing of erroneous rulings.
• 7.3 Lack of capability or robustness: The AI models failed to perform reliably by hallucinating non-existent legal precedents and facts when applied to legal drafting tasks.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by ChatGPT and Perplexity generating hallucinated legal citations and quotes, which was an unintentional and unexpected outcome during post-deployment use by court staff.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The tools used were general-purpose chatbots (ChatGPT and Perplexity) which are subject to transparency obligations. The report notes staff used 'ChatGPT to perform legal research' and 'Perplexity as a drafting assistant'.

AI system and alleged parties

• AI system: ChatGPT, Perplexity (OpenAI, Perplexity AI)
• AI purpose: Writing Assistant; Judicial Decision Support
• Behaviour type: Assistant
• Alleged developer: Perplexity AI, OpenAI
• Alleged deployer: Unnamed law school intern (D.N.J.), Unnamed law clerk (S.D. Miss.), United States District Court for the Southern District of Mississippi, United States District Court for the District of New Jersey, Julien Neals, Henry Wingate
• Alleged harmed parties: United States District Court for the Southern District of Mississippi, United States District Court for the District of New Jersey, Plaintiffs and defendants in Jackson Federation of Teachers et al. v. Mississippi State Board of Education, Plaintiffs and defendants in CorMedix Securities Litigation, Julien Neals, Judicial integrity, Henry Wingate, Epistemic integrity

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: Yes, the report explicitly describes the generation and docketing of factually inaccurate court documents containing fake quotes and fictional litigants.

Directly caused: The AI systems generated fabricated legal citations, nonexistent quotes, and false facts that were published in official court opinions.

Indirectly caused: One of the faulty opinions was submitted as 'supplemental authority' in another class-action lawsuit, spreading the false information further into the legal ecosystem.

Inferred additional harm: If undetected, these fabricated cases could have been cited by other lawyers or indexed by legal databases, permanently corrupting the legal record.

People affected

• Occurrences reported: 2
• People reportedly harmed: 10
• People reportedly exposed: 100

Potential causes

Management

• Inadequate Judicial Oversight: Judges failed to adequately scrutinize documents issued under their names.
• Lack of Initial Transparency: Judges did not immediately disclose AI use when errors were first flagged.
• Busy Docket Management Pressures: High workload pressured chambers to use AI tools as drafting assistants.

Technology

• Generative AI Hallucinations: AI tools confidently invented facts, fake quotes, and nonexistent cases.
• Use of Unverified AI Assistants: Perplexity and ChatGPT were used for drafting and legal research.

Human Factors

• Overreliance on AI Tools: Staff relied on AI output without independently verifying the cited cases.
• Lack of Staff Training: Clerk and intern lacked understanding of generative AI hallucination risks.
• Negligent Verification: Staff failed to read and verify the actual cases cited by the AI.

Process and Methods

• Inadequate Review Processes: Draft opinions were docketed without undergoing a final review process.
• Lack of Written Chamber Policies: Chambers relied on verbal understandings rather than written AI bans or rules.
• Premature Document Docketing: Drafts containing AI errors were mistakenly published before final approval.

Regulatory Environment

• Absence of Binding AI Rules: Federal courts lacked formalized, decisive measures to prevent AI misuse.
• Insufficient Interim Guidance: Interim guidance only suggested considering AI disclosure, lacking mandates.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed, coming from reputable news sources, and contain direct admissions from the federal judges involved, along with specific details of the cases and the AI tools used.

U.S. federal court staff used ChatGPT and Perplexity to draft rulings, resulting in the docketing of documents with hallucinated legal citations. While this compromised judicial decision-making processes, the national security impact is minor and is being managed through administrative guidance and internal reviews.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding the integration of generative AI in government functions rather than an active crisis.
• Autonomy: Human-controlled. AI tools were used as assistants for drafting and research, with humans retaining final decision-making and docketing authority.
• Novelty: Evolved capability. While AI hallucinations in legal filings have occurred before, this represents an evolved threat where federal judicial chambers directly published the errors.

Impact by dimension

• Physical security: Negligible. No physical systems, infrastructure, or human safety were impacted or threatened by this incident.
• Information security: Negligible. No classified information, intelligence capabilities, or systematic information warfare campaigns were involved or compromised.
• Sovereignty: Minor. Compromise of federal judicial drafting processes occurred when hallucinated AI outputs were docketed. However, the impact was minor and managed through standard administrative retractions, policy updates, and a Senate inquiry.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or economic manipulation occurred as a result of this incident.
• Societal stability: Negligible. No mass surveillance, systemic discrimination, or threat to societal stability occurred.