Meta AI Hallucinates Harassment Allegations Against New York Politicians

Meta's AI chatbot in Facebook Messenger falsely accused multiple state lawmakers of sexual harassment, fabricating incidents, investigations, and consequences that never occurred. These fabricated stories, discovered by City & State, sparked outrage among the affected lawmakers and raised concerns about the reliability of the chatbot. Meta acknowledged the errors and committed to ongoing improvements.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The chatbot generated and spread completely fabricated and false allegations of sexual harassment against individuals, which constitutes false or misleading information.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The incident stems from the AI system's technical limitation of hallucinating realistic-sounding but entirely fabricated facts.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk was caused by the Meta AI system hallucinating false information, which was an unexpected and unintended outcome of its deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3. Limited Risk: The system is a chatbot, which falls under limited risk and requires specific transparency obligations to ensure users are informed they are interacting with an AI.

AI system and alleged parties

• AI system: Meta AI (Meta)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: Meta, Facebook users
• Alleged deployer: Meta
• Alleged harmed parties: New York lawmakers, Meta, Kristen Gonzalez, Facebook users, Clyde Vanel

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the chatbot generated and spread fabricated sexual harassment allegations.

Directly caused: Meta AI generated detailed, false stories of sexual misconduct, investigations, and professional punishments against over a dozen lawmakers.

Epistemic

Reported: Yes, the reports explicitly discuss the chatbot's hallucinations as spreading misinformation and disinformation.

Directly caused: Meta AI fabricated detailed stories of sexual harassment allegations, investigations, and consequences (like being stripped of committee assignments) for over a dozen lawmakers.

Inferred additional harm: Widespread use of the chatbot could lead to users believing false allegations about public officials, eroding shared consensus reality.

People affected

• Occurrences reported: 1
• People reportedly harmed: 13
• People reportedly exposed: 15

Potential causes

Management

• Premature Technology Deployment: Deploying generative AI features before resolving known hallucination issues.
• Reliance on User Disclaimers: Shifting the safety and verification burden to users via general warnings.

Technology

• Generative AI Hallucinations: The underlying LLM inherently fabricates details without verification.
• Platform-Dependent Behavior: Inconsistent safety behavior across mobile, desktop, and login states.

Data Inputs

• Lack of Verified Source Links: The model generated text without grounding in verifiable online sources.
• Sensitive Prompt Vulnerability: Prompts with names and sensitive terms easily triggered false narratives.

Human Factors

• User Prompting with Sensitive Terms: Users combined politician names with sensitive words, triggering safety gaps.

Process and Methods

• Inadequate Content Filtering: Lack of robust guardrails to block generation of defamatory content.
• Ineffective Multi-Platform Testing: Testing failed to catch variations in desktop versus mobile app responses.

Regulatory Environment

• Absence of Defamation Regulations: Current laws do not specifically address AI-generated defamatory content.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are consistent, detailed, and describe a well-documented phenomenon of LLM hallucination with direct quotes from affected lawmakers and Meta representatives.
• Ambiguities identified: None of significance.
• Alternative interpretations: None.

Meta's AI chatbot fabricated detailed, false sexual harassment allegations against several New York state lawmakers. While causing personal distress and highlighting generative AI misinformation risks, the incident was an unintentional technical hallucination with negligible direct impact on national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The immediate incident has been identified and addressed by Meta, representing an ongoing strategic concern regarding AI-generated misinformation rather than an active crisis.
• Autonomy: Full autonomy. The Meta AI chatbot generated and outputted the false allegations autonomously in response to user prompts without human review or intervention.
• Novelty: Evolved capability. AI hallucinations and misinformation are established threats, but the fabrication of highly specific, localized political misconduct details represents an evolved capability concern.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were impacted or threatened by the chatbot hallucinations.
• Information security: Minor. While the AI fabricated false narratives about public officials, this was an unintentional technical hallucination rather than a coordinated foreign intelligence or information warfare campaign.
• Sovereignty: Minor. The incident targeted state-level lawmakers, causing temporary reputational concern, but did not disrupt core government functions, elections, or state sovereignty.
• Economic security: Negligible. No financial systems, strategic technologies, or critical supply chains were compromised or targeted in this incident.
• Societal stability: Minor. The chatbot generated defamatory content about individuals, which could minorly erode trust in public information, but did not cause large-scale civil unrest or systemic human rights violations.