Purported Gemini-Generated AI Images Reportedly Claimed U.S. Delta Force Soldiers Were Captured by the IRGC

Images purportedly generated by Gemini were reportedly circulated on X and Facebook in multiple languages claiming U.S. soldiers had been captured by the Iranian Revolutionary Guard Corps during the war in Iran. AFP said the images contained Gemini markers and other visual inconsistencies. Their circulation allegedly contributed to wartime misinformation and risked distorting public understanding of the conflict, including reported amplification by Jill Stein.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The AI systems were used to generate realistic fake imagery of an active conflict to conduct a large-scale disinformation campaign aimed at manipulating public opinion.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI models failed to robustly enforce their own safety filters, allowing users to generate realistic images of active warzones despite policies prohibiting it.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was primarily driven by human actors intentionally generating and spreading deceptive AI-generated images to manipulate public opinion during an active conflict.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes AI systems being used to generate synthetic media and deepfakes of warzones, which are subject to transparency obligations to ensure users are aware the content is AI-generated.

AI system and alleged parties

• AI system: Gemini, Grok (Google, xAI)
• AI purpose: Image Generation; Deepfake Image Generation
• Behaviour type: Assistant
• Alleged developer: Google
• Alleged deployer: Unknown pro-Iranian disinformation actors, Unknown disinformation actors, Tayyaba Usmani
• Alleged harmed parties: social media users, People seeking information about the war in Iran, National security and intelligence stakeholders, General public, Epistemic integrity

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Substantial
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports describe the creation and viral spread of deceptive, realistic images of active combat and captured soldiers.

Directly caused: AI models Gemini and Grok generated realistic images of US and Iranian troops shooting each other and captured military personnel.

Indirectly caused: These deceptive images were shared widely across social media platforms like X and Facebook, reaching thousands of users.

Democracy

Reported: Yes, the reports describe how the disinformation campaign aimed to manipulate public opinion and political perceptions surrounding the war.

Indirectly caused: The viral spread of fake military captures and propaganda on social media platforms undermines public trust and misleads citizens during a geopolitical crisis.

Epistemic

Reported: Yes, the reports explicitly describe the proliferation of AI-generated fake images that mislead the public and erode shared reality regarding the war.

Directly caused: AI systems generated highly realistic fake images of military events, such as captured US Delta Force troops and downed B-2 bombers, which were presented as real.

Indirectly caused: The fake images flooded social media platforms, fooling prominent figures and ordinary users, thereby polluting the information ecosystem and complicating public understanding of the conflict.

Inferred additional harm: Widespread exposure to these realistic fakes likely causes long-term erosion of trust in digital media and visual evidence of real-world conflicts.

People affected

• Occurrences reported: 1
• People reportedly exposed: 120000000

Potential causes

Management

• Prioritizing Engagement: Platforms prioritized engagement through emotional and polarising content.
• Flawed Policy Enforcement: Platforms failed to enforce their own rules against creating warzone images.

Technology

• Inconsistent Guardrails in Grok: Grok allowed generation of war zone images despite rules prohibiting it.
• Imperfect AI Image Generation: Generated images had visual anomalies like extra limbs and warped hands.
• Inability to Detect Bypasses: Grok failed to block prompt bypasses on different user accounts.

Data Inputs

• Permissive Input Prompts: System accepted prompts depicting active combat without triggering blocks.

Human Factors

• Rapid Mobile Consumption: Users consume media quickly on small screens, reducing verification time.
• Sharing Without Verification: Users and prominent figures reposted fake images without checking origins.
• Intentional Disinformation: Some users posted fake images to promote a pro-Iran war narrative.

Process and Methods

• Inconsistent Account Blocking: Safeguards blocked one account but allowed another on the same device.
• Delayed Platform Mitigation: Content moderation and community notes applied after images went viral.
• Lack of Automatic Warnings: Grok provided realistic combat images without issuing warnings.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, detailed, and consistent accounts of how Gemini and Grok were used to generate fake images of the conflict in Iran, including specific examples, user reactions, and platform responses. The role of the AI systems is explicitly documented, and the nature of the misinformation is verified by fact-checkers.

Commercial generative AI models were exploited to generate realistic, deceptive imagery of US military losses in Iran. The rapid viral spread of these fakes on social media represents a substantial information warfare incident that successfully misled millions of viewers and highlighted vulnerabilities in AI safety filters during active geopolitical conflicts.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: United States
• Other affected: Iran
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Near-term. The threat of active conflict disinformation remains a highly active, near-term concern requiring ongoing monitoring and platform intervention.
• Autonomy: Human-controlled. The AI systems acted as passive tools, generating the deceptive images only when directly and intentionally prompted by human users.
• Novelty: Evolved capability. This represents an evolved capability where consumer-grade text-to-image models are easily manipulated to bypass safety filters and produce highly convincing war propaganda.

Impact by dimension

• Physical security: Negligible. The incident was entirely limited to digital disinformation and did not involve kinetic threats, physical harm, or critical infrastructure disruption.
• Information security: Substantial. Generative AI was used to create highly realistic fake imagery of military operations (captured US troops and downed bombers) during an active conflict, successfully deceiving millions of people and prominent political figures to manipulate public perception.
• Sovereignty: Minor. The spread of fake military captures by a former presidential candidate represents a minor disruption to political discourse, but did not directly compromise government decision-making or sovereign operations.
• Economic security: Negligible. No significant economic infrastructure, financial systems, or strategic technologies were targeted or compromised during this incident.
• Societal stability: Minor. The incident eroded public trust and polluted the information ecosystem during a geopolitical crisis, though it did not result in systemic human rights abuses or widespread civil unrest.