Unrestricted AI Avatar Tool Accidentally Released by TikTok Permits Recitation of Hitler Quotes and Other Harmful Speech

TikTok accidentally released an internal version of its AI digital avatar tool without safeguards, allowing users to generate videos where avatars could recite harmful content, including quotes from Hitler. The tool, meant for advertisers, was accessible to personal accounts and lacked the watermark indicating AI-generated content. TikTok has since removed the tool and acknowledged the problem.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The leaked AI tool allowed the generation of highly toxic content, including hate speech, white supremacist slogans, and dangerous instructions to drink bleach.

Additional risk subdomains

• 3.1 False or misleading information: The tool was used to generate civic disinformation telling people to vote on the wrong day and health misinformation about drinking bleach.
• 7.3 Lack of capability or robustness: The incident involved a technical error where an internal testing version lacking standard safety guardrails and watermarks was exposed to the public.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by a human technical error that accidentally exposed an unmoderated internal testing version of an AI tool to the public.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3. Limited Risk: The system is an AI video generator that creates synthetic content (deepfakes/digital avatars), which is subject to transparency obligations such as watermarking under the EU AI Act.

AI system and alleged parties

• AI system: Symphony Digital Avatars internal testing version (TikTok)
• AI purpose: Deepfake Video Generation; Social Media Content Generation
• Behaviour type: Tool
• Alleged developer: TikTok
• Alleged deployer: TikTok
• Alleged harmed parties: TikTok users, General public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 5

Potential causes

Management

• Inadequate Release Oversight: Lack of strict controls allowed internal testing tools to leak publicly.

Technology

• No Generation Guardrails: The internal tool lacked active content filtering during video generation.
• Missing AI Watermarks: Generated videos lacked watermarks identifying them as AI-created.

Data Inputs

• Unfiltered Script Inputs: The system accepted any text script input without prior validation.

Human Factors

• Accidental Link Exposure: An employee mistakenly posted a link to an internal testing tool.

Process and Methods

• Weak Environment Isolation: Internal testing environments were accessible via public internet links.
• Deficient Access Control: Personal accounts bypassed the Ads Manager restriction to use the tool.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent details about the nature of the technical error, the specific content generated during testing, and TikTok's response. There is no major conflict between the sources.
• Ambiguities identified: It is not fully clear how many other users accessed the tool or if any generated videos were actually posted publicly by others.

TikTok accidentally leaked an unmoderated internal version of its Symphony Digital Avatars tool, allowing users to generate deepfakes containing hate speech, medical misinformation, and civic disinformation without watermarks. While the tool was quickly removed and no widespread dissemination occurred, the incident highlights the risks of accidental exposure of powerful synthetic media generation capabilities.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: TikTok

Threat characteristics

• Imminence: Long-term. The immediate security issue was resolved when TikTok removed the tool, representing a long-term strategic concern regarding deepfake tool security rather than an active crisis.
• Autonomy: Human-controlled. The AI system functions as a tool that generates video based on direct human text inputs, with humans making all decisions regarding content.
• Novelty: Evolved capability. Represents an evolved risk vector where a major corporate platform accidentally releases its unmoderated internal deepfake engine to the public.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety systems were targeted or impacted by this incident.
• Information security: Minor. The tool allowed generation of hate speech and disinformation without watermarks, but it was an accidental leak and the generated content was not widely disseminated in an active information operation.
• Sovereignty: Minor. Although the tool was used to generate civic disinformation regarding voting dates, this was done by journalists in a testing capacity and did not disrupt actual electoral processes.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or significant economic impacts were reported from this accidental leak.
• Societal stability: Minor. The tool enabled generation of toxic content and hate speech, but there is no evidence of large-scale societal disruption or human rights violations occurring prior to its removal.