ChatGPT Provided Non-Existent Citations and Links when Prompted by Users

When prompted about providing references, ChatGPT was reportedly generating non-existent but convincing-looking citations and links, which is also known as "hallucination".

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated false and misleading information, including fabricated obituaries of a living person and non-existent academic citations, which can lead to inaccurate beliefs.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The model's tendency to hallucinate facts and generate fake references represents a lack of capability and robustness in critical information retrieval tasks.
• 5.1 Overreliance and unsafe use: Users relied on the chatbot to generate programming answers and academic references without verifying their correctness, leading to the pollution of platforms like Stack Overflow.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The generation of fabricated information and fake obituaries was an unintended failure mode of the deployed ChatGPT model.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot generating text, which falls under transparency obligations to ensure users are aware they are interacting with an AI.

AI system and alleged parties

• AI system: ChatGPT 3.5 (OpenAI)
• AI purpose: Chatbot; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI
• Alleged harmed parties: ChatGPT users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Substantial
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report explicitly describes privacy concerns regarding OpenAI using personal information from the internet without consent.

Directly caused: The AI processed and generated false personal data about PERSON_006 (incorrect birth year, birth location, and fake death) without consent.

Inferred additional harm: Millions of individuals' personal data was scraped and processed to train the model, leading to potential widespread privacy violations and inaccurate profiling.

Epistemic

Reported: The report explicitly describes extensive epistemic harm, including AI-driven fabrication, fake evidence, fabricated citations, and defamation-by-hallucination.

Directly caused: The AI system generated completely fabricated obituaries, fake URLs, non-existent academic citations, and false quotes from CEOs.

Indirectly caused: Stack Overflow had to ban ChatGPT responses due to a high volume of inaccurate, authoritative-sounding answers polluting the platform.

Inferred additional harm: Widespread erosion of trust in online information as users struggle to distinguish between factual content and highly convincing AI hallucinations.

People affected

• Occurrences reported: 5
• People reportedly harmed: 3
• People reportedly exposed: 100

Potential causes

Management

• Hasty Public Deployment: Releasing unaligned models to the public driven by commercial pressure.
• Poor Incident Response: Management failed to respond to reports of false death claims.

Technology

• Next-Word Prediction Objective: Model predicts likely words instead of verifying factual accuracy.
• Authoritative Phrasing Bias: Model generates false information with a highly confident and formal tone.
• Lack of Abstract Reasoning: Model struggles with abstract concepts, leading to logical errors.

Data Inputs

• Uncurated Internet Training Data: Training on unverified web data leads to learning falsehoods and biases.
• Ingestion of Personal Data: Processing personal data without consent leads to privacy violations.
• Lack of Source Citation Data: Training data lacks structured citations, causing fabricated references.

Human Factors

• User Over-reliance on Hype: Users accept outputs as absolute truth without verifying correctness.
• Malicious User Manipulation: Users deliberately coax or jailbreak the model to output misinformation.
• Lack of Verification by Users: Users publish AI outputs without checking facts or credentials.

Process and Methods

• Inadequate Alignment Methods: RLHF fails to completely eliminate hallucinations and toxic outputs.
• Inadequate Output Filtering: Lack of robust real-time filters allows false obituaries to be displayed.
• Biased Labeler Preferences: RLHF training aligns with a narrow, English-speaking demographic.

Regulatory Environment

• Lack of AI Safety Regulations: No legally binding frameworks exist to enforce model safety standards.
• Weak Data Privacy Enforcement: Regulators fail to prevent LLMs from harvesting and outputting personal data.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, first-hand accounts of the AI's hallucination behavior, backed by technical explanations from OpenAI's own documentation regarding InstructGPT and RLHF.
• Ambiguities identified: None significant; the behavior of the model is well-documented and consistent across different reports.
• Alternative interpretations: The incident could be viewed purely as a technical limitation (7.3) rather than misinformation (3.1), but the primary real-world impact described is the generation of false information.

The widespread public release of ChatGPT highlighted the system's tendency to generate highly convincing but entirely fabricated information, including fake obituaries and academic citations. While causing minor epistemic harm and individual distress, the incident represents a technological limitation rather than a coordinated national security threat.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The tendency of LLMs to hallucinate represents an ongoing strategic and technical challenge rather than an immediate national security crisis.
• Autonomy: Human-controlled. The AI acts as an assistant generating text in response to direct human prompts, with humans retaining control over how the output is used.
• Novelty: Evolved capability. The incident represents a significant advancement in the scale and convincing nature of AI-generated misinformation, building on existing language model limitations.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic impacts, or critical infrastructure compromises were reported in connection with this incident.
• Information security: Minor. The model generated false information and fake citations, leading to minor pollution of technical forums like Stack Overflow, but no coordinated information warfare or intelligence compromise occurred.
• Sovereignty: Negligible. No impact on state authority, government decision-making, electoral systems, or constitutional processes was reported.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or significant threats to national economic security were reported.
• Societal stability: Minor. The incident caused minor psychological distress to affected individuals due to fake obituaries and raised privacy concerns regarding data scraping, but did not threaten large-scale societal stability.