Celebrities' Deepfake Voices Abused with Malicious Intent

Voices of celebrities and public figures were deepfaked using voice synthesis for malicious intents such as impersonation or defamation, and were shared on social platforms such as 4chan and Reddit.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The incident involved the generation and distribution of highly toxic, racist, transphobic, and violent audio clips using cloned voices of celebrities on 4chan.

Additional risk subdomains

• 4.3 Fraud, scams, and targeted manipulation: The cloning of public figures' voices without consent to depict them saying offensive things is a form of targeted manipulation and harassment.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human users intentionally exploiting ElevenLabs' beta platform to generate harmful and non-consensual deepfake audio clips after the tool was deployed.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system generates deepfakes (AI-generated content), which under the EU AI Act are subject to specific transparency obligations to ensure users are informed of their synthetic nature.

AI system and alleged parties

• AI system: ElevenLabs voice-cloning tool, VALL-E (ElevenLabs, Microsoft)
• AI purpose: Voice Generation
• Behaviour type: Tool
• Alleged developer: ElevenLabs
• Alleged deployer: Reddit users, ElevenLabs users, 4chan users
• Alleged harmed parties: public figures, celebrities

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports explicitly describe the generation and distribution of toxic, racist, transphobic, and violent audio clips on 4chan.

Directly caused: Users generated audio clips of Emma Watson reading 'Mein Kampf', David Attenborough making racist remarks, and other clips containing violent, homophobic, and transphobic statements.

Indirectly caused: The distribution of these clips on 4chan exposed forum members and the wider public to highly offensive and abusive synthetic media.

Inferred additional harm: It is likely that many more unmentioned toxic clips were generated and shared across various online communities during the beta period.

People affected

• Occurrences reported: 1
• People reportedly harmed: 4
• People reportedly exposed: 100000

Potential causes

Management

• Feverish Rush to Market: Pressure to stand out in a noisy AI hype cycle led to cutting corners.
• Inadequate Risk Assessment: Management failed to anticipate and prevent obvious misuse vectors.

Technology

• High-Fidelity Voice Cloning: Technology allows highly realistic voice clones from short audio samples.
• Low Barrier to Entry: Easy-to-use web interface allowed rapid generation of deepfake audio.

Data Inputs

• Minimal Audio Sample Requirements: Only one minute of clean audio was needed to clone a target voice.
• Unverified Voice Ownership: No check to ensure users had rights or consent to the voice sample used.

Human Factors

• Malicious Intent by Online Trolls: 4chan users intentionally exploited the tool to generate hateful content.
• Naive Assumptions of User Intent: Founders made naive assumptions about the goodwill of early adopters.

Process and Methods

• Anonymous Free Tier Sign-Ups: Lack of identity or payment verification allowed anonymous abuse.
• Absence of Pre-Release Safeguards: Beta launched without content filtering or voice verification tools.

Regulatory Environment

• Lack of Solid Legislation: Flimsy or absent laws governing deepfake audio and non-consensual media.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from multiple reputable sources consistently describe the same core event: ElevenLabs' beta platform being used by 4chan trolls to clone celebrity voices for toxic content, and the company's subsequent response. The details are clear and uncontradicted.
• Ambiguities identified: None of significance; the role of ElevenLabs' software is clearly established by both the company's own statements and investigative reporting.

Anonymous users on 4chan exploited ElevenLabs' beta voice-cloning platform to generate and distribute non-consensual, hateful, and defamatory audio clips of celebrities and public figures, including US President Joe Biden. While the incident caused reputational and psychological harm, its national security impact remains minor as there was no evidence of coordinated state-sponsored information warfare or disruption to government functions.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, United Kingdom
• Alleged perpetrator: 4chan users

Threat characteristics

• Imminence: Long-term. The immediate platform misuse was addressed by ElevenLabs, but the underlying capability of accessible voice-cloning represents an ongoing strategic concern.
• Autonomy: Human-controlled. The AI voice-cloning platform acted strictly as a tool, generating audio based on direct human text inputs and voice samples.
• Novelty: Evolved capability. While deepfakes existed, the high-quality, low-barrier, and rapid generation of cloned voices represented a significant advancement in accessible generative AI capabilities.

Impact by dimension

• Physical security: Negligible. No physical infrastructure or kinetic systems were targeted or affected in this incident.
• Information security: Minor. Generation of deepfake audio of public figures, including US President Joe Biden, by internet trolls on 4chan represents a minor information security threat without evidence of coordinated state-sponsored information warfare.
• Sovereignty: Minor. Although the voice of US President Joe Biden was cloned, the incident was limited to online trolling on 4chan and did not disrupt core government functions or decision-making.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or critical supply chain disruptions occurred as a result of this incident.
• Societal stability: Minor. The incident caused localized psychological and reputational harm to targeted individuals through toxic synthetic media, but did not trigger large-scale civil unrest or systemic human rights violations.