Microsoft Copilot Designer Reportedly Generated Inappropriate AI Images

A Microsoft engineer reported that Copilot Designer, an AI image generator, creates content depicting sex, violence, bias, and more. Despite raising concerns and suggesting improvements, the tool remains public, prompting a letter to the FTC.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system generated and exposed users to toxic, violent, and sexually inappropriate content from benign prompts.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The model lacked the robustness to safely filter prompts and prevent the generation of harmful images.
• 6.5 Governance failure: Microsoft failed to implement recommended safety disclosures or update the app's age rating despite internal warnings.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The generation of inappropriate and harmful images from benign prompts was an unintended failure of the deployed AI system.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes Copilot Designer as a generative AI tool that creates AI-generated content (images), which falls under transparency obligations for AI-generated content.

AI system and alleged parties

• AI system: Copilot Designer (Microsoft)
• AI purpose: Image Generation; Visual Art Generation
• Behaviour type: Tool
• Alleged developer: Microsoft
• Alleged deployer: Microsoft
• Alleged harmed parties: minors, General public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the generation of toxic and inappropriate content by the AI system.

Directly caused: The AI system directly generated images depicting sexual objectification, violence, underage drinking, drug use, and conspiracy theories from benign prompts.

Inferred additional harm: It is highly likely that other public users generated and potentially shared similar toxic or inappropriate images before safeguards were updated.

Epistemic

Reported: The report mentions that the tool produced conspiracy theories and political bias, which can contribute to epistemic harm.

Directly caused: The AI system generated images reflecting conspiracy theories and political bias.

Inferred additional harm: The generation of conspiracy-related imagery could contribute to the spread of misinformation among users, though the scale of this impact is unquantified.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Prioritizing Competition Over Safety: Company rushed product to market, ignoring internal safety warnings.
• Silencing Internal Whistleblowers: Legal team ordered employee to delete posts raising concerns.
• Failure to Pause Public Access: Management rejected recommendations to remove the tool from public use.

Technology

• Unsafe Generative Model: DALL-E model generates harmful content from benign prompts.
• Inadequate Prompt Safeguards: System fails to prevent harmful image generation from benign inputs.

Data Inputs

• Unfiltered Training Data: Model was trained on data that allows generating inappropriate imagery.

Human Factors

• Unsupervised Use by Children: Children use the tool unsupervised for school or education projects.

Process and Methods

• Inadequate App Rating: Android app rated 'E for Everyone' instead of 'Mature 17+'.
• Lack of Product Disclosures: Product lacks clear warnings or disclosures about potential harmful outputs.

Regulatory Environment

• Lack of Regulatory Standards: Absence of government mandates enforcing safety standards on AI tools.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly details the specific prompts used, the resulting inappropriate outputs, and the internal whistleblowing process by a Microsoft employee. The facts regarding the AI system's behavior and the company's response are well-documented.
• Ambiguities identified: None significant, though the exact scale of public exposure and whether any external children actually generated these images is not fully detailed.

A Microsoft whistleblower reported that Copilot Designer generates harmful, violent, and sexually objectified images from benign prompts. While presenting minor concerns for societal stability and information integrity due to the generation of conspiracy theories and political bias, the incident is primarily a corporate safety and regulatory matter with negligible direct national security impact.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing product safety and regulatory concern rather than an active, imminent national security crisis.
• Autonomy: Human-controlled. The AI operates as a tool generating images directly from user prompts without independent planning or decision-making capabilities.
• Novelty: Evolved capability. Demonstrates an evolved capability of generative AI models to bypass safety filters, building on established text-to-image vulnerabilities.

Impact by dimension

• Physical security: Negligible. No physical infrastructure, kinetic targeting, or human safety systems were affected by the image generation tool.
• Information security: Minor. The tool produced conspiracy theories and political bias, showing a minor vulnerability to information manipulation, though no active exploitation by foreign adversaries was reported.
• Sovereignty: Negligible. No disruption to state authority, electoral systems, or core government decision-making processes was observed.
• Economic security: Negligible. Represents a corporate governance and safety failure for Microsoft, but does not threaten national economic stability or strategic technological advantage.
• Societal stability: Minor. Generates harmful content including violent and sexually objectified images from benign prompts, posing minor risks to public safety and child welfare.