AI-Generated Voices Amplify Conspiracy Theories on TikTok

NewsGuard has identified 17 TikTok accounts that have been using AI-generated voices to advance and amplify conspiracy theories and false claims beginning in June 2023. By September 25, 2023, these accounts had amassed over 336 million views and over 14.5 million likes. Videos include baseless claims involving public figures such as Barack Obama, Oprah Winfrey, and Jamie Foxx.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The incident involved a coordinated network of 17 TikTok accounts using AI-generated voices to conduct a large-scale disinformation campaign that garnered over 336 million views.

Additional risk subdomains

• 3.1 False or misleading information: The AI system was used to generate and spread highly realistic false claims and conspiracy theories about public figures.
• 4.3 Fraud, scams, and targeted manipulation: The creators cloned the voices of specific public figures like Barack Obama to fabricate statements and manipulate public perception.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human actors intentionally using ElevenLabs' voice cloning technology to spread disinformation after the tool was deployed.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is used to generate deepfakes and synthetic audio, which falls under Risk Level 3 due to transparency and disclosure requirements.

AI system and alleged parties

• AI system: ElevenLabs text-to-speech, unspecified AI voice generator (ElevenLabs)
• AI purpose: Voice Generation; Social Media Content Generation
• Behaviour type: Tool
• Alleged developer: ElevenLabs
• Alleged deployer: TikTok users, TikTok user @ynewstv2023, TikTok user @e.news.tv, TikTok user @drphilshowtv, TikTok user @d.news.tv
• Alleged harmed parties: Yahoo! News, TikTok, Phil McGraw, Oprah Winfrey, Joan Rivers, Jamie Foxx, General public, E! News, Barack Obama

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Severe, indirect Severe
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report explicitly describes the creation and spread of malicious conspiracy theories using AI-generated voices.

Directly caused: The network of accounts generated videos pushing false claims, such as Barack Obama being connected to his chef's death and Oprah Winfrey being a sex trader.

Inferred additional harm: It is likely that similar malicious content was generated and spread across other platforms like YouTube, Instagram, and Facebook, as noted in the report.

Democracy

Reported: Yes, the report describes how audio fakes threaten to turbocharge political disinformation ahead of elections.

Indirectly caused: The proliferation of realistic AI-generated voices of political figures like Barack Obama undermines public trust and democratic discourse by making it easy to fabricate statements.

Inferred additional harm: The widespread use of such tools could lead to systemic erosion of trust in democratic processes and elections.

Epistemic

Reported: Yes, the report explicitly describes epistemic harm through the spread of AI-generated conspiracy theories and false information.

Directly caused: The network of 17 TikTok accounts generated videos that received 336 million views and 14.5 million likes, spreading false claims about Barack Obama, Oprah Winfrey, Jamie Foxx, and the Maui fires.

Indirectly caused: The comments on the videos show that some viewers believed the false claims, such as users commenting that Obama had his chef killed.

Inferred additional harm: The scale of these views suggests that millions of users may have been misled or had their beliefs manipulated by the fabricated audio.

People affected

• Occurrences reported: 1
• People reportedly harmed: 4
• People reportedly exposed: 336000000

Potential causes

Management

• Reactive Moderation Policies: Platforms acted only after NewsGuard flagged the viral misinformation.
• Delayed Access Safeguards: ElevenLabs only restricted voice cloning after widespread public misuse.

Technology

• Easy Voice Cloning Tools: ElevenLabs allows creating lifelike audio clones with a few clicks.
• Flawed AI Voice Detectors: Detectors fail when background music or distortion is added to the audio.
• Inherent Audio Flagging Challenges: Synthetic audio is uniquely difficult to flag consistently compared to video.

Data Inputs

• Unrestricted Voice Uploads: Users could upload any voice recording to generate highly realistic clones.
• Baseless Text-to-Speech Scripts: Conspiracy texts were fed into AI models to generate realistic narration.

Human Factors

• Deceptive Creator Intent: Bad actors intentionally used AI to spread rumors and gain followers.
• User Susceptibility to Audio Fakes: Viewers believed the AI voice claims, as audio is sticky in memory.

Process and Methods

• Lack of Proactive AI Labeling: Creators failed to use TikTok's voluntary AI disclosure labels.
• Inadequate Platform Detection: Social media platforms lacked robust automated tools to detect AI audio.

Regulatory Environment

• Lack of Industry Standards: No binding regulations exist to enforce labeling of synthetic media.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from NYT and NewsGuard provide highly detailed, consistent, and verified information about the network of accounts, the specific AI tool used (ElevenLabs), the scale of views (336 million), and the specific false narratives spread.
• Ambiguities identified: None significant; the role of ElevenLabs and the actions of the creators are clearly documented.
• Alternative interpretations: None; this is a clear-cut case of malicious use of voice-cloning technology for disinformation.

A network of 17 TikTok accounts used ElevenLabs' voice cloning technology to run a massive disinformation campaign, generating over 336 million views with fake audio of public figures like Barack Obama. While the scale of exposure was substantial, the incident was managed through platform content moderation and did not directly compromise critical infrastructure, active government operations, or national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic concern regarding the proliferation of realistic synthetic media rather than an active, immediate crisis.
• Autonomy: Human-controlled. The AI system acted purely as a tool, generating synthetic voice outputs based on direct human text inputs and configurations.
• Novelty: Evolved capability. While voice cloning and disinformation are established threats, the scale, ease of use, and realism enabled by consumer AI tools represent an evolved capability.

Impact by dimension

• Physical security: Negligible. No physical security or critical infrastructure threats were reported in connection with this incident.
• Information security: Minor. The incident involved a large-scale AI-enabled disinformation campaign targeting public figures with 336 million views, but it was managed through standard social media platform moderation.
• Sovereignty: Minor. While the campaign targeted a former U.S. President, it did not compromise active government decision-making, electoral systems, or core state sovereignty.
• Economic security: Negligible. No strategic technology theft, financial system manipulation, or critical supply chain impacts were reported.
• Societal stability: Minor. The campaign reached millions of users, spreading false narratives that eroded public trust, but did not lead to civil unrest or systemic human rights violations.