Microsoft AI Is Alleged to Have Generated Violent Imagery of Minorities and Public Figures

Microsoft’s AI Image Creator, integrated with Bing and Windows Paint, produced disturbingly violent and graphic images featuring members of minority groups and public figures like Joe Biden and Pope Francis.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system generated highly graphic, violent, and disturbing images of decapitations and flayed skin, exposing users to toxic and unsafe content.

Additional risk subdomains

• 4.1 Disinformation, surveillance, and influence at scale: Extremist groups on 4chan used the tool to coordinate large-scale propaganda and disinformation campaigns to inflame geopolitical tensions.
• 7.3 Lack of capability or robustness: The AI's safety guardrails failed to perform robustly, allowing simple semantic bypasses to generate prohibited content.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk of generating graphic violence and propaganda was an unexpected and unintentional outcome of deploying the image generation system.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3. Limited Risk: AI systems that pose a moderate risk and require specific transparency obligations... Examples: Chatbots, AI-generated content such as deepfakes. The report describes Bing Image Creator as an 'AI text-to-image generator' used to create 'deepfake' images of politicians and celebrities.

AI system and alleged parties

• AI system: DALL-E 3 3 (OpenAI)
• AI purpose: Image Generation; Deepfake Image Generation
• Behaviour type: Assistant
• Alleged developer: Microsoft
• Alleged deployer: Windows Paint, Microsoft, Bing users, Bing, AI Image Creator
• Alleged harmed parties: Sikh people, President Joe Biden, Pope Francis, Navajo people, Minorities, Hillary Clinton, General public, Donald Trump

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Substantial
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Minor
• Democracy: direct Minor, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports extensively describe the generation of graphic violence, Nazi imagery, and racist/antisemitic memes.

Directly caused: The AI generated realistic images of politicians and celebrities with their necks sliced open and ethnic minorities with flayed skin.

Indirectly caused: 4chan users generated and disseminated Nazi propaganda, antisemitic memes, and inflammatory content regarding the Israel-Gaza conflict across social media.

Inferred additional harm: A significant volume of toxic, racist, and violent synthetic media circulating on the internet, polluting online spaces and social media platforms.

Differential treatment

Reported: Yes, the reports mention the generation of violent images targeting specific ethnic-minority groups and women.

Directly caused: The AI generated images of Sikh and Navajo people with their internal organs spilling out of flayed skin, representing targeted racialized violence.

Indirectly caused: Extremists used the tool to spread explicitly racist and antisemitic memes, promoting hate speech against minority groups.

Inferred additional harm: Increased harassment and marginalization of ethnic and religious minorities due to the proliferation of AI-generated hate propaganda.

Civil rights

Reported: Yes, the reports describe the generation of hate speech and harassment targeting specific groups.

Directly caused: The generation of graphic, dehumanizing imagery of ethnic minorities violates their right to be free from racial harassment and discrimination.

Indirectly caused: The coordinated dissemination of Nazi and antisemitic propaganda on 4chan and social media undermines civil rights and promotes hate-fueled hostility.

Inferred additional harm: Potential escalation of real-world hate crimes or systemic discrimination against targeted groups inspired by the viral spread of these violent images.

Democracy

Reported: Yes, the reports note the risk to upcoming general elections due to the ease of creating deepfakes of politicians.

Directly caused: The AI generated graphic deepfakes of political figures like Joe Biden and Donald Trump with mortal wounds.

Indirectly caused: 4chan users created and shared political propaganda designed to inflame public opinion and manipulate discourse around the Israel-Gaza conflict.

Inferred additional harm: Erosion of public trust in democratic institutions, political candidates, and the integrity of electoral processes due to the proliferation of realistic, highly disturbing political deepfakes.

Psychological

Reported: Yes, the reports describe the images as realistic enough to upset people and disturbing to users.

Directly caused: The whistleblower reported feeling 'demoralized' by Microsoft's rejection of his safety reports.

Indirectly caused: Public figures and members of ethnic minorities were depicted in highly graphic, violent ways, which is highly distressing and abusive.

Inferred additional harm: Widespread distress and anxiety among users, particularly children or families using Windows Paint/Bing, who may have accidentally encountered these graphic decapitation images.

Epistemic

Reported: Yes, the reports mention the images look realistic enough to mislead people and create deepfakes of politicians.

Directly caused: The AI generated highly realistic fake photos of real people in violent scenarios.

Indirectly caused: The dissemination of these realistic deepfakes on social networks misleads the public about real-world events and individuals.

Inferred additional harm: Degradation of the shared information ecosystem, making it harder for the public to distinguish between real and synthetic media.

People affected

• Occurrences reported: 1
• People reportedly harmed: 5
• People reportedly exposed: 100000

Potential causes

Management

• Prioritizing Market Speed Over Safety: Company rushed to integrate generative AI into apps to not miss the boat.
• Shifting Blame to Users: Management framed safety failures as unauthorized user misuse.
• Inadequate Safety Team Investment: Declined executive interviews and delayed fixing known gore-generation bugs.

Technology

• Flawed Output Guardrails: System failed to block prompts using euphemisms like red corn syrup.
• Inconsistent Partner Integration: Microsoft failed to use safety blocks present in partner OpenAI's DALL-E 3.

Data Inputs

• Unfiltered Internet Training Data: AI was trained on data from the internet containing violent imagery.

Human Factors

• Adversarial User Exploitation: Users actively designed prompts to bypass guardrails for memetic warfare.
• Malicious 4chan Campaigns: Far-right groups coordinated to flood the internet with racist AI images.

Process and Methods

• Flawed Bug Bounty Triaging: Security team rejected prompt bypass submissions as non-vulnerabilities.
• Ignored User Feedback Channels: Reports submitted via feedback and concern forms were repeatedly ignored.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the 'kill prompt' vulnerability, the specific types of graphic images generated, and the exploitation of the tool by 4chan users. The facts are well-documented across multiple independent sources, with clear quotes from Microsoft and OpenAI spokespeople confirming the issues.
• Ambiguities identified: The exact wording of the 'kill prompt' is intentionally withheld by the journalists for safety reasons, and the precise technical mechanism of Microsoft's guardrail failure is not fully detailed.
• Alternative interpretations: The incident could be viewed primarily as user misuse rather than a system failure, though the reports strongly emphasize the failure of Microsoft's safety guardrails and bug bounty program.

Extremist groups exploited vulnerabilities in Microsoft's Bing Image Creator to bypass safety guardrails, generating graphic political deepfakes and targeted hate propaganda. This incident highlights the evolved threat of generative AI being leveraged for information warfare and societal destabilization.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: United States
• Other affected: Canada
• Alleged perpetrator: Extremist groups on 4chan's '/pol/' board

Threat characteristics

• Imminence: Near-term. The ongoing generation of political deepfakes and propaganda poses a near-term threat to public trust and upcoming electoral processes.
• Autonomy: Human-controlled. The AI acted as an assistant, generating images strictly based on human-engineered 'kill prompts' and bypass techniques.
• Novelty: Evolved capability. The incident represents a significant advancement in the realism of generated propaganda and the sophistication of semantic guardrail bypasses.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic targeting capabilities were affected or threatened during this incident.
• Information security: Substantial. Extremists actively exploited the tool to conduct large-scale propaganda campaigns, generating graphic deepfakes of political figures and inflaming geopolitical tensions.
• Sovereignty: Minor. While graphic deepfakes of political leaders were created, there was no direct compromise of core government operations, electoral systems, or constitutional processes.
• Economic security: Negligible. The incident did not involve strategic technology theft, financial system attacks, or critical supply chain disruptions.
• Societal stability: Substantial. The tool was used to generate highly graphic, racialized violence and Nazi/antisemitic propaganda, threatening social cohesion and promoting harassment of minority groups.