Google AI Reportedly Delivering Confidently Incorrect and Harmful Information

Google's AI search engine has reportedly been providing users with confidently incorrect and often harmful information. Reports highlight numerous inaccuracies, including misleading health advice and dangerous cooking suggestions. For example, it has falsely claimed Barack Obama as the first Muslim U.S. President, reflecting fringe conspiracy theories, or recommending that glue can be an ingredient in pizza.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated and spread false and misleading information, such as fabricated historical facts and dangerous recipes, presenting them as factual summaries.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The system failed to perform reliably, hallucinating answers and failing to distinguish satirical content from factual information.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the Gemini AI system generating incorrect and hallucinated outputs after being deployed to the public.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The AI Overviews feature acts as a text-generation system and chatbot-like interface, which falls under the category of systems requiring transparency so users know they are interacting with AI-generated content.

AI system and alleged parties

• AI system: AI Overviews (Google)
• AI purpose: Question Answering; Content Search
• Behaviour type: Assistant
• Alleged developer: Google
• Alleged deployer: Google
• Alleged harmed parties: Google users, General public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Substantial, indirect Substantial
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic or dangerous content created and spread directly by the incident.

Directly caused: The AI directly generated instructions to mix chlorine bleach and white vinegar (creating toxic chlorine gas) and falsely stated that Barack Obama is Muslim, repeating a racist conspiracy theory.

Epistemic

Reported: The report explicitly describes epistemic harm caused directly by the incident through AI-driven misinformation and fabrication.

Directly caused: The AI generated numerous false summaries, including that Obama was the first Muslim president, that glue should be added to pizza, and that rocks should be eaten daily.

Indirectly caused: The dissemination of these falsehoods online eroded trust in Google Search as an authoritative source of information.

Inferred additional harm: Millions of users who saw these overviews may have been temporarily misled or had their trust in online information further degraded.

People affected

• Occurrences reported: 1
• People reportedly exposed: 2000000000

Potential causes

Management

• Pressure to Compete with Rivals: Management rushed the rollout to keep pace with Microsoft and OpenAI.
• Prioritizing Speed over Safety: Executive decision to move fast and skip critical safety-evaluation steps.
• Overconfidence in AI Readiness: Leaders promoted the tech as ready to 'do the googling' at a large scale.

Technology

• Large Language Model Hallucinations: LLMs guess mathematically likely words but cannot reason or verify facts.
• Inability to Recognize Satire: The AI cannot distinguish between factual data and jokes or satirical posts.
• Lack of Fact-Checking Mechanisms: The system cites sources but cannot evaluate if the source material is true.

Data Inputs

• Training on Unfiltered Web Data: LLM was trained on open web data containing falsehoods and satire.
• Sourcing from Online Forums: AI retrieved answers from old, joking Reddit comments and forum posts.
• Sparse Information on Rare Queries: For uncommon queries, the model relied on low-quality, isolated sources.

Human Factors

• User Coaxing and Manipulation: Users actively tried to trick the AI into generating incorrect answers.
• High Public Expectations: Users expect search engines to be authoritative, unlike conversational LLMs.
• Overreliance on Automated Outputs: Presenting AI summaries at the top of search results encourages blind trust.

Process and Methods

• Insufficient Scale Testing: Pre-launch testing failed to catch errors that appeared at public scale.
• Ineffective Triggering Refinements: Initial system lacked filters to block satire or forum posts on health topics.
• Rapid Rollout Strategy: Google deployed the experimental feature to all US users simultaneously.

Regulatory Environment

• Lack of AI Safety Standards: No formal regulatory oversight for deploying generative AI in search engines.
• Unclear Publisher Liability: Lack of legal clarity on whether AI search is a creator or an aggregator.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed, and widely-publicized examples of the AI's failures, along with official statements from Google confirming the issues and detailing their mitigation steps. There is no significant disagreement among the sources regarding what happened.

Google's deployment of AI Overviews resulted in widespread model hallucinations, including dangerous chemical advice and factual errors. While exposing millions of users to misinformation, the incident represents an unintentional consumer product failure rather than a directed national security threat, resulting in minor overall impact.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, United Kingdom
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing, systemic challenge regarding LLM reliability and public trust rather than an active, immediate crisis.
• Autonomy: Human-supervised. The AI system generated search summaries autonomously, but operated within Google's managed infrastructure with human capability to intervene and patch.
• Novelty: Established threat. Large language model hallucinations and the failure to distinguish satire from fact are well-documented technical issues.

Impact by dimension

• Physical security: Minor. The AI generated dangerous instructions such as mixing chemicals to create toxic chlorine gas, but this was unintentional and did not target critical infrastructure.
• Information security: Minor. The system generated and spread false information and historical inaccuracies, but this was due to model hallucination rather than an active foreign disinformation campaign.
• Sovereignty: Negligible. No impact on state authority, government decision-making, or core constitutional processes.
• Economic security: Minor. Highlighted deployment vulnerabilities and caused reputational harm to a major US tech firm, but did not involve strategic theft or systemic economic damage.
• Societal stability: Minor. Exposed millions of users to potentially harmful advice and misinformation, slightly degrading epistemic trust, but did not cause civil unrest or systematic rights violations.