National Company Law Tribunal Reportedly Relied on AI-Hallucinated Precedents in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd

In Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., India's National Company Law Tribunal (NCLT) reportedly admitted insolvency proceedings against Essel Infraprojects after relying on six purported precedents later found to be fabricated or materially inaccurate. The appellate tribunal upheld the ruling, but the Supreme Court later invalidated both decisions and sent the insolvency petition back to the NCLT for a new ruling.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated fabricated legal precedents and false quotes, leading the tribunal to make a judicial decision based on false beliefs, which caused material legal harm to the affected company.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The NCLT judges and NCLAT appellate authority relied on AI-generated legal research without verifying the authenticity of the cited precedents.
• 7.3 Lack of capability or robustness: The underlying AI system failed to perform reliably, producing severe hallucinations and fabricating non-existent case law.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by an AI system generating hallucinated, non-existent legal precedents, which was an unexpected and unintended outcome of its deployment for legal research.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The AI system was used in the administration of justice to assist in judicial decision-making, which has significant implications for fundamental rights (such as the right to a fair trial) and public interests.

AI system and alleged parties

• AI system: unspecified AI
• AI purpose: Judicial Decision Support; Question Answering
• Behaviour type: Assistant
• Alleged developer: Large language model developers, AI research tool developers
• Alleged deployer: National Company Law Tribunal, Courts
• Alleged harmed parties: Pooja Ramesh Singh, Parties to insolvency proceedings, Judicial integrity, Essel Infraprojects Ltd, Epistemic integrity

Harm severity

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: Yes, the report explicitly describes the fabrication of six legal citations and false propositions of law.

Directly caused: The AI system generated six fabricated legal citations and false propositions of law which were integrated into an official judicial order.

Indirectly caused: The NCLAT upheld the NCLT order without independently verifying the citations, propagating the false information further up the judicial hierarchy.

Inferred additional harm: N/A

People affected

• Occurrences reported: 2
• People reportedly harmed: 2
• People reportedly exposed: 2

Potential causes

Management

• Lack of AI Risk Training: Management failed to train judicial officers on AI hallucination risks.
• Inadequate Judicial Oversight: No oversight mechanism existed to monitor AI use in tribunal research.

Technology

• AI Hallucination: AI system generated fake, non-existent, and hallucinated legal precedents.
• Lack of Fact-Checking in AI: AI system lacked mechanisms to cross-reference real legal databases.

Data Inputs

• Unverified AI Search Outputs: The tribunal used raw, unverified AI outputs as factual judicial research.

Human Factors

• Judicial Overreliance on AI: Tribunal relied on AI capabilities, infiltrating intellectual work ethics.
• Failure to Verify Citations: The tribunal failed to manually check the existence of the cited judgments.

Process and Methods

• Inadequate Citation Verification: No standard procedure existed to verify self-sourced judicial citations.
• Deficient Appellate Review: NCLAT upheld the order without independently verifying the cited authorities.

Regulatory Environment

• Absence of AI Guidelines in Law: No regulatory framework governed the use of AI tools in judicial drafting.
• Lack of Misconduct Penalties: No clear rules existed classifying unverified AI citations as misconduct.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, detailed, and consistent information about the specific hallucinated judgments, the court's reaction, and the legal consequences. The role of the AI system (generating fabricated precedents) is explicit, and the resulting harm to the judicial process is well-documented.
• Ambiguities identified: The specific AI model or platform used by the tribunal is not identified in the report.
• Alternative interpretations: None. The facts of the hallucination and its impact on the court ruling are undisputed.

An Indian judicial tribunal relied on six AI-hallucinated legal precedents in an insolvency ruling, which was subsequently overturned by the Supreme Court. The incident highlights minor risks to government functions and judicial integrity from unverified AI usage, but lacks broader national security implications.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: India
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding the integration of AI tools in state legal systems rather than an active crisis.
• Autonomy: Human-supervised. The AI system assisted with legal research, but human judges retained final decision-making authority and responsibility for verification.
• Novelty: Evolved capability. While AI hallucination is an established issue, its integration into official judicial rulings represents an evolved risk to institutional processes.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety were threatened or impacted by this incident.
• Information security: Negligible. No intelligence compromise, classified data theft, or systematic hostile information warfare campaigns were identified.
• Sovereignty: Minor. The incident affected a judicial tribunal's decision-making process, but was corrected by the Supreme Court, representing a manageable issue within standard procedures.
• Economic security: Negligible. No strategic technology theft, systemic financial system attack, or loss of technological competitive advantage occurred.
• Societal stability: Minor. While the incident briefly compromised the integrity of a judicial determination for the affected party, it did not cause large-scale threats to social cohesion.