Gemini and Grok Reportedly Misidentified Authentic Minab School-Strike Graveyard Photo as Unrelated Disaster Imagery

Gemini and Grok reportedly gave false provenance claims about an authentic photo of graves prepared for victims of the Minab school strike, identifying it as disaster imagery from Turkey or Indonesia. Researchers verified the photo's location through satellite imagery and related visual evidence.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI systems generated and spread authoritative but entirely false information about a real-world photograph, leading to inaccurate beliefs and undermining factual consensus.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The models failed to perform reliably, hallucinating facts and fabricating sources when faced with real-time information verification tasks.
• 5.1 Overreliance and unsafe use: Users are increasingly relying on AI summaries for critical news and information, treating probabilistic chatbots as factual 'truth boxes'.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The false information was generated by deployed AI systems (Gemini and Grok) behaving in an unexpected and inaccurate manner while attempting to answer user queries.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The systems involved are conversational chatbots and AI-generated content generators, which are subject to specific transparency obligations under the EU AI Act.

AI system and alleged parties

• AI system: Gemini, Grok (Google, xAI)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: xAI, Google
• Alleged deployer: xAI, Google
• Alleged harmed parties: Shajareh Tayyebeh Primary School community, Shajareh Tayyebeh Primary School, Open-source investigators, Minab school-strike victims and families, Minab Hermud Cemetery, Iranian schoolchildren, Human rights investigators, Fact-checkers, Epistemic integrity, Educational communities

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Minor, indirect Substantial
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the creation and spread of AI-generated slop and viral falsehoods.

Directly caused: Gemini and Grok directly generated false narratives, hallucinated facts, and fabricated sources denying the authenticity of the Minab cemetery photo.

Civil rights

Reported: The report explicitly describes challenges for those investigating human rights abuses and the risk of atrocities being denied.

Indirectly caused: The propagation of false AI summaries wastes the time of human rights investigators, diverting resources away from documenting the impact of the war on civilians.

Inferred additional harm: The systematic denial of real-world atrocities by AI systems could severely hinder international human rights documentation and future legal accountability efforts.

Democracy

Reported: The report explicitly describes risks to public information ecosystems as people increasingly rely on AI summaries for news.

Indirectly caused: The widespread dissemination of inaccurate AI summaries undermines the public's ability to access reliable news, weakening informed democratic citizenship.

Inferred additional harm: The erosion of public trust in journalism and authentic media due to a deluge of AI-generated misinformation could severely damage democratic discourse.

Psychological

Reported: The report explicitly describes psychological distress and disrespect experienced by the families of the deceased.

Indirectly caused: The families of the killed children face deep disrespect and emotional distress when AI systems are used online to claim that the tragic event did not happen.

Inferred additional harm: Grieving families of conflict victims globally may suffer secondary trauma and distress as AI-generated denials of real-world atrocities circulate widely.

Epistemic

Reported: The report explicitly describes extensive epistemic harm, including hallucinated facts, nonsense analysis, and the risk of real atrocities being dismissed as fake.

Directly caused: Gemini and Grok fabricated false origins, dates, and locations for an authentic photograph, providing non-existent sources and dead-end links.

Indirectly caused: The deluge of AI falsehoods muddies the waters, making it difficult for the public to distinguish between real events and synthetic content, thereby eroding shared reality.

Inferred additional harm: A broader, long-term loss of consensus reality where authentic visual evidence of historical events is routinely doubted and dismissed as AI-generated.

People affected

• Occurrences reported: 1
• People reportedly harmed: 102
• People reportedly exposed: 1000

Potential causes

Management

• Premature public deployment: Tools were rolled out globally despite high documented error rates.
• Lack of platform accountability: Companies ignore inquiries and rely on disclaimers to shift blame.

Technology

• Probabilistic nature of LLMs: Models generate likely words rather than verifying factual truth.
• Lack of visual analysis capability: AIs fail to analyze actual image pixels, relying on text associations.
• Authoritative tone of false outputs: AIs present incorrect facts with high confidence and fake sources.

Data Inputs

• Hallucinated source citations: AIs invent nonexistent links and news reports to back up claims.
• Misassociation of historical data: AIs link current war images to old Covid or earthquake photos.

Human Factors

• Overreliance on AI summaries: Users rely on AI summaries instead of checking primary news sources.
• Perception of AI omniscience: Users trust AI as an objective 'truth box' free of human bias.
• Increased skepticism of real media: Skepticism leads people to dismiss authentic atrocity images as fake.

Process and Methods

• Ineffective chatbot correction loop: Prompting AI for corrections leads to further hallucinated errors.
• Inadequate output verification: Platforms lack automated checks to verify AI-generated assertions.

Regulatory Environment

• Absence of liability for AI errors: Fine print disclaimers shield platforms from legal consequences.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, direct evidence of the AI systems' failures, including specific hallucinated responses, the real-world context of the photograph, and the direct impacts on researchers and victims' families. There is no significant ambiguity regarding the AI's role or the nature of the errors.

Google Gemini and xAI Grok chatbots generated authoritative but false provenance information regarding an authentic photo of a mass burial site in Iran. This unintentional misinformation hindered human rights investigations and highlighted systemic risks of AI hallucinations eroding public trust in factual evidence during active conflicts, though it posed minimal direct threat to national security.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Iran, United States, Israel
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing, systemic vulnerability regarding AI truthfulness rather than an active, time-sensitive national security crisis.
• Autonomy: Human-supervised. The AI chatbots generated and presented false narratives autonomously in response to user queries without real-time human oversight of the output accuracy.
• Novelty: Established threat. AI model hallucination and the generation of authoritative-sounding false information is a well-established threat in LLM deployment.

Impact by dimension

• Physical security: Negligible. No physical security threat, kinetic attacks, or critical infrastructure compromise occurred as a result of the chatbot hallucinations.
• Information security: Minor. Minor impact. Unintentional AI-generated misinformation diluted authentic war imagery, complicating open-source intelligence verification but without representing an active, coordinated foreign intelligence operation.
• Sovereignty: Negligible. No significant threat to state authority, electoral integrity, or core government decision-making processes was reported.
• Economic security: Negligible. No impact on financial systems, strategic technologies, supply chains, or economic security was identified.
• Societal stability: Minor. Minor impact. The incident impeded human rights documentation and risked facilitating atrocity denial, affecting societal trust in authentic evidence, but did not trigger mass civil unrest.