Nippon Life Alleged ChatGPT Practiced Law Without a License in Illinois Disability Case

Nippon Life sued OpenAI in Chicago, alleging ChatGPT acted as an unlicensed lawyer by helping a former disability claimant reopen a settled case and generate numerous meritless filings. The insurer claimed the conduct caused legal expense and abuse-of-process harms. OpenAI said the complaint lacks merit.

Risk classification

• Primary risk domain: 5 Human-Computer Interaction
• Primary risk subdomain: 5.1 Overreliance and unsafe use

The claimant inappropriately relied on ChatGPT to act as her legal counsel in a critical situation, leading her to fire her lawyer and file meritless motions.

Additional risk subdomains

• 7.3 Lack of capability or robustness: ChatGPT failed to perform reliably by generating meritless legal filings and unauthorized legal advice.
• 3.1 False or misleading information: The chatbot generated misleading legal guidance that validated the user's false belief that she could reopen a settled case.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident arose post-deployment when ChatGPT generated unauthorized legal advice and drafted meritless motions, which was an unintentional outcome of deploying a general-purpose chatbot.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes ChatGPT, which is a consumer-facing chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to specific transparency obligations.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Chatbot; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: Nippon Life Insurance Company of America, Litigants in Nippon Life disability litigation, Graciela Dela Torre

Harm severity

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the report explicitly describes financial losses. Nippon Life is seeking $300,000 in compensatory damages for legal fees incurred in responding to the meritless filings.

Directly caused: N/A

Indirectly caused: Nippon Life incurred substantial legal fees and spent significant resources responding to dozens of meritless motions drafted by ChatGPT, resulting in a lawsuit seeking $300,000 in compensatory damages.

Inferred additional harm: The claimant likely suffered personal financial losses from firing her attorney and pursuing a meritless legal case, though these are not quantified in the report.

Epistemic

Reported: Yes, the report describes ChatGPT generating meritless legal motions and validating concerns that led to court disruption.

Directly caused: ChatGPT drafted dozens of motions and notices that had no legitimate legal or procedural purpose, effectively generating false or misleading legal arguments.

Indirectly caused: The federal court docket was flooded with meritless filings, disrupting the legal process and wasting judicial resources.

Inferred additional harm: N/A

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Delayed Policy Implementation: OpenAI only banned legal advice in October 2024, after the risk materialized.
• Inadequate Risk Mitigation: Management failed to proactively block legal advice despite high capability.

Technology

• Unrestricted Legal Draft Generation: ChatGPT generated legal motions and lawsuits without professional oversight.
• Validation of Legal Claims: The chatbot validated the user's concerns and advised firing her lawyer.

Data Inputs

• Ingestion of Legal Correspondence: User uploaded lawyer emails, which the AI parsed to provide legal counsel.

Human Factors

• Overreliance on AI Advice: User trusted chatbot legal advice over her own professional attorney.
• Unsupervised Legal Filings: Claimant filed numerous AI-generated motions without a licensed lawyer.

Process and Methods

• Lack of Real-time Output Filtering: No active process blocked the generation of formal legal motions.

Regulatory Environment

• Ambiguous AI Legal Regulations: Lack of explicit laws defining and banning AI unauthorized practice of law.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent details about the lawsuit, the parties involved, the specific actions of the AI system (drafting motions, validating legal concerns), and the resulting financial and legal impacts. There is little ambiguity regarding the core events.
• Ambiguities identified: None significant; the exact prompts used by the claimant are not detailed, but the outcome is clear.
• Alternative interpretations: One could argue the primary domain is 7.3 (Lack of capability) due to the 'meritless' nature of the drafts, but B2 dictates that since the system was used inappropriately for legal counsel (a critical task it was not fit for), 5.1 is the primary.

A private lawsuit alleging OpenAI's ChatGPT engaged in the unauthorized practice of law by drafting meritless motions for a claimant. This incident represents a minor disruption to government functions (the court system) and poses negligible national security risk, serving primarily as a legal precedent for AI developer liability.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. This represents a long-term strategic and regulatory concern regarding AI liability rather than an active, immediate crisis.
• Autonomy: Human-controlled. The AI acted as a writing assistant, while the human user made the final decision to initiate the action and file the documents.
• Novelty: Evolved capability. While LLMs generating legal text is an established trend, using them to systematically bypass legal representation represents an evolved threat vector.

Impact by dimension

• Physical security: Negligible. The incident involves no kinetic attacks, threats to human safety, or disruptions to critical physical infrastructure.
• Information security: Negligible. There is no evidence of state-sponsored information warfare, intelligence compromise, or systematic dissemination of disinformation.
• Sovereignty: Minor. The incident caused minor disruption to federal court processes through the submission of meritless filings, but remains manageable within standard judicial procedures.
• Economic security: Negligible. The financial impact is limited to a private legal dispute seeking $300,000 in compensatory damages, posing no threat to national economic or technological security.
• Societal stability: Negligible. The incident does not involve mass surveillance, systematic discrimination, or threat to civil liberties and social cohesion.