Grok AI Reportedly Generates Offensive and Violent Images Without Proper Safeguards

Elon Musk’s Grok AI, launched on X, generated offensive and violent images without adequate safety controls. The AI produced deepfakes of public figures like Taylor Swift, Kamala Harris, and Alexandria Ocasio-Cortez, as well as copyrighted characters such as Mickey Mouse in inappropriate scenarios. Despite claiming adherence to certain content guidelines, Grok's outputs included politically charged and explicit imagery

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The Grok-2 image generator produced highly inappropriate, violent, and sexualized images of public figures and copyrighted characters, exposing users to toxic and abusive content.

Additional risk subdomains

• 4.3 Fraud, scams, and targeted manipulation: The tool was used to generate non-consensual sexualized and humiliating deepfakes targeting specific female public figures.
• 3.1 False or misleading information: The generator produced hyper-realistic, unwatermarked images of politicians in fabricated scenarios, which could mislead the public.

Causal factors

• Entity: AI
• Intent: Intentional
• Timing: Post-deployment

The incident occurred post-deployment when users intentionally generated offensive and copyrighted images using the newly released Grok-2 tool, which xAI deliberately deployed with minimal safety guardrails.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is an AI chatbot capable of generating deepfakes, which falls under the category of AI systems that pose a moderate risk and require specific transparency obligations (such as notifying users that content is AI-generated).

AI system and alleged parties

• AI system: Grok-2 2 (xAI)
• AI purpose: Image Generation; Chatbot
• Behaviour type: Assistant
• Alleged developer: xAI, X (Twitter)
• Alleged deployer: xAI
• Alleged harmed parties: Taylor Swift, Nintendo, Kamala Harris, Joe Biden, Donald Trump, Disney, Alexandria Ocasio-Cortez

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Substantial
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Minor, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Minor, indirect Substantial
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the reports mention potential copyright infringement of characters owned by Nintendo and Disney, and advertisers fleeing X due to hateful content being displayed next to their ads.

Indirectly caused: X faced potential loss of advertising revenue as ads were displayed next to offensive generated images, and copyright holders face potential devaluation of intellectual property.

Inferred additional harm: Substantial indirect financial losses to X from lost ad revenue and potential legal defense costs, as well as brand damage to intellectual property owners, though unquantified.

Malicious content

Reported: Yes, the reports describe a flood of offensive, violent, and sexually explicit AI-generated images.

Directly caused: Grok-2 generated images of politicians in lingerie, a prophet holding a bomb, characters doing drugs, and violent scenes like mass shootings and 9/11 attacks.

Indirectly caused: These toxic images were widely shared across X and other social media platforms, exposing a large audience to inappropriate content.

Inferred additional harm: Millions of users on X and other platforms were likely exposed to these toxic and offensive deepfakes.

Differential treatment

Reported: Yes, female public figures were disproportionately targeted with sexualized deepfakes.

Directly caused: Female public figures (Taylor Swift, Kamala Harris, AOC) were targeted with sexualized images (lingerie), reflecting gender-based harassment.

Inferred additional harm: Widespread generation of non-consensual sexual deepfakes disproportionately harms women, reinforcing systemic gender-based harassment online.

Civil rights

Reported: Yes, experts note that the unmitigated creation of these images violates civil rights and constitutes sexual harassment.

Directly caused: The generation of non-consensual sexualized deepfakes of women violates their right to personal dignity and freedom from harassment.

Inferred additional harm: Widespread abuse of image generators without guardrails threatens the civil rights of individuals to control their own likeness and be free from targeted harassment.

Democracy

Reported: Yes, the reports mention the tool being used to create misinformation about political figures, and five secretaries of state calling on Musk to fix Grok's election-related falsehoods.

Directly caused: Grok-2 generated false images of political figures (e.g., Trump lying down with blackened hands after an explosion, politicians flying planes into the WTC) and previously spread misinformation about Kamala Harris's ballot eligibility.

Indirectly caused: The proliferation of unwatermarked, hyper-realistic political deepfakes undermines public trust in visual evidence and democratic processes.

Inferred additional harm: The unchecked generation of political deepfakes during election periods poses a severe threat to democratic integrity by misleading voters at scale.

Psychological

Reported: Yes, the reports describe public figures being targets of online harassment, degradation, and sexualized deepfakes.

Directly caused: Public figures like Taylor Swift, Kamala Harris, and Alexandria Ocasio-Cortez were targeted with sexualized and degrading deepfakes, causing psychological distress and online harassment.

Inferred additional harm: It is highly likely that other public figures and private individuals experienced severe distress and anxiety due to non-consensual deepfakes generated by users during this period.

Epistemic

Reported: Yes, the reports describe the generation of hyper-realistic but entirely fabricated images leading to the spread of false information.

Directly caused: Grok-2 generated unwatermarked, realistic-looking deepfakes of public figures in compromising or false scenarios (e.g., Trump kissing Musk, Harris making a dosa).

Indirectly caused: The lack of watermarks or origin tags on Grok's outputs puts downstream consumers at risk of misinterpreting deepfakes as genuine, eroding shared reality.

Inferred additional harm: Widespread epistemic degradation as users lose the ability to distinguish real photos from AI-generated ones, leading to a general erosion of trust in digital media.

People affected

• Occurrences reported: 1
• People reportedly harmed: 6
• People reportedly exposed: 10000

Potential causes

Management

• Prioritizing Innovation Over Safety: Management prioritized rapid feature release over robust safety guardrails.
• Unrestrictive Free Speech Philosophy: Leadership favored an unrestrictive approach to avoid censorship.

Technology

• Lack of Content Moderation Filters: Grok-2 lacked standard filters to block misleading or offensive images.
• Absence of Image Watermarking: No watermarks were applied to identify images as AI-generated.
• Immature AI Model Deployment: Deploying an immature model led to unpredictable and harmful outputs.

Data Inputs

• Unrestricted Prompt Inputs: Users could input prompts featuring real people and copyrighted characters.
• Use of Copyrighted Likenesses: The model was trained or allowed to generate copyrighted characters.

Human Factors

• User Intent to Create Misinformation: Users actively exploited the lack of filters to create offensive fakes.
• Widespread Lack of Media Literacy: Downstream consumers easily believe realistic AI fakes to be real.

Process and Methods

• Premature Model Release: The model was released in beta without rigorous safety testing.
• Lax Policy Enforcement on X: Enforcement of safety policies became lax after the platform takeover.
• No Post-Generation Takedown Policy: Lack of mechanisms to easily take down generated images shared elsewhere.

Regulatory Environment

• Absence of Clear AI Regulations: Lack of government frameworks allowed unregulated AI deployment.
• Delay in Passing the No Fakes Act: Legislation to protect likenesses from deepfakes was not yet enacted.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the release of Grok-2, its lack of safety guardrails, and the specific types of offensive and copyrighted images generated by users. Multiple experts and legal professionals are quoted, confirming the technical and legal implications of the tool's deployment.
• Ambiguities identified: None. The reports are clear about the AI system's capabilities, the lack of guardrails, and the resulting outputs.
• Alternative interpretations: None.

The release of xAI's Grok-2 with minimal safety guardrails allowed users to generate unwatermarked, hyper-realistic deepfakes of political leaders and public figures. This event represents a substantial information security concern, particularly regarding the potential for election-related disinformation and targeted harassment, highlighting the evolving challenges of unregulated generative AI tools.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: United States
• Alleged perpetrator: Individuals

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic and capability concern regarding information integrity rather than an immediate kinetic crisis.
• Autonomy: Human-controlled. The AI system generates images based entirely on direct user prompts and does not act or deploy content autonomously.
• Novelty: Evolved capability. While deepfake technology is an established threat, the deliberate release of a high-profile generator completely lacking standard safety guardrails and watermarks represents an evolved capability.

Impact by dimension

• Physical security: Negligible. No physical threats or critical infrastructure compromises were reported in connection with this incident.
• Information security: Substantial. The tool enabled the generation of unwatermarked, hyper-realistic deepfakes of political leaders in violent or compromising scenarios, creating significant potential for systematic disinformation.
• Sovereignty: Minor. While five US secretaries of state raised concerns over election-related falsehoods, there was no reported direct compromise of government decision-making or electoral systems.
• Economic security: Negligible. Economic impacts were limited to corporate advertising losses for X and potential copyright issues for intellectual property holders, with no threat to strategic national industries.
• Societal stability: Minor. The tool was used to generate non-consensual sexualized deepfakes targeting female public figures, representing a form of gender-based harassment, but did not trigger widespread civil unrest.