Google Reports Alleged Gemini-Generated Terrorism and Child Exploitation to Australian eSafety Commission

Google reported to Australia's eSafety Commission that it received 258 complaints globally about AI-generated deepfake terrorism content and 86 about child abuse material made with its Gemini AI. The regulator called this a "world-first insight" into AI misuse. While Google uses hash-matching to detect child abuse content, it lacks a similar system for extremist material.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The incident involves the generation of deepfake terrorist material and child sexual abuse material, which directly constitutes exposure to toxic, harmful, and illegal content as defined under sub-domain 1.2.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system lacked robust safeguards to prevent the generation of highly harmful and illegal content like extremist deepfakes.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident involves human users intentionally utilizing the deployed Gemini AI system to generate prohibited and harmful content, representing a post-deployment misuse of the technology.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The report describes Gemini as an AI program used to generate deepfakes and other synthetic content, which falls under the transparency and disclosure obligations of Limited Risk systems.

AI system and alleged parties

• AI system: Gemini (Google)
• AI purpose: Deepfake Image Generation; Deepfake Video Generation
• Behaviour type: Assistant
• Alleged developer: Google
• Alleged deployer: Google
• Alleged harmed parties: Victims of online radicalization, Victims of deepfake terrorism content, Victims of deepfake child abuse, Google Gemini users, General public of Australia, General public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Minor

Malicious content

Reported: Yes, the report explicitly describes the generation of deepfake terrorist material and child abuse material using Gemini.

Directly caused: Google received 258 reports of Gemini being used to generate deepfake terrorist or violent extremist content, and 86 reports of it being used to create child exploitation or abuse material.

Inferred additional harm: It is likely that more instances of toxic content were generated but went unreported by users during this period.

Child sexual exploitation and abuse

Reported: Yes, the report explicitly describes 86 user reports alleging Gemini was used to generate child exploitation or abuse material.

Indirectly caused: The report notes 86 global user reports of Gemini being used to create child exploitation or abuse material between April 2023 and February 2024.

Inferred additional harm: It is highly likely that additional undetected instances of AI-generated CSAM occurred, further contributing to child exploitation.

People affected

• Occurrences reported: 344

Potential causes

Management

• Inconsistent Safety Implementation: Management deployed safety tools for child abuse but not terrorism.

Technology

• Lack of Terrorist Hatch-Matching: Hatch-matching was not used to identify and remove terrorist material.
• Inadequate Generative Guardrails: Gemini AI allowed generation of deepfake terrorism and abuse content.

Human Factors

• Exploitation by Malicious Users: Users actively exploited Gemini to produce harmful and illegal content.

Process and Methods

• Insufficient Safeguard Testing: Lack of effective testing of safeguards to prevent harmful generations.
• Ineffective Complaint Verification: Google did not verify how many user complaints were policy violations.

Regulatory Environment

• Evolving Global AI Regulations: Regulators globally are still calling for better AI safety guardrails.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the statistics of user complaints (258 for terrorism, 86 for CSAM) and Google's regulatory reporting to the Australian eSafety Commission. The roles of the AI system (Gemini) and the developer (Google) are explicitly stated, leaving little ambiguity about the nature of the incident.
• Ambiguities identified: The report does not specify how many of the user complaints were verified as actual policy violations or confirmed generations of illegal material.

Google reported that its Gemini AI was used to generate deepfake terrorist content and child exploitation material. While highlighting gaps in AI safety guardrails, particularly the lack of extremist content filtering, the incident represents a minor national security threat manageable through standard regulatory and law enforcement mechanisms.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident reflects an ongoing strategic vulnerability in AI safety guardrails rather than an active, imminent crisis.
• Autonomy: Human-controlled. The AI system acted as a tool directly prompted and controlled by human users to generate specific illegal content.
• Novelty: Evolved capability. While generating extremist and child abuse material is an established threat, using advanced multimodal AI models to easily generate such content represents an evolved capability.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic actions, or critical infrastructure compromises were reported in connection with this incident.
• Information security: Minor. Users generated deepfake terrorist and extremist content, representing potential information operations, but there is no evidence of coordinated state-sponsored campaigns.
• Sovereignty: Negligible. The incident does not involve threats to state authority, electoral systems, or core government decision-making processes.
• Economic security: Negligible. While regulatory compliance issues and potential fines are noted, there is no threat to strategic industries, critical supply chains, or national technological advantage.
• Societal stability: Minor. The generation of child abuse and extremist material represents a serious violation of human rights, but the impact is manageable through standard regulatory and law enforcement channels.