TikTok AI System Used to Amplify Election Disinformation by Foreign Networks

AI-generated misinformation on TikTok, driven by foreign networks, has flooded the platform with false narratives about the 2024 U.S. presidential election. Thousands of videos spreading political lies were identified, potentially influencing millions of users. Despite TikTok’s efforts to remove these accounts, the AI-driven disinformation campaign continues to challenge the integrity of the election.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The incident involves coordinated foreign networks using AI-generated content to conduct large-scale disinformation campaigns aimed at manipulating public opinion and political processes in the U.S.

Additional risk subdomains

• 3.2 Pollution of information ecosystem and loss of consensus reality: The coordinated campaign flooded TikTok with AI-generated misinformation, polluting the platform's information ecosystem.
• 7.3 Lack of capability or robustness: TikTok's moderation and recommendation systems lacked the robustness to proactively detect and prevent the spread of these automated campaigns.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was driven by human malicious actors intentionally deploying AI-generated content and automated accounts to exploit social media platforms post-deployment.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The incident involves the creation and dissemination of AI-generated content, including synthetic voices and manipulated media, which falls under transparency obligations for AI-generated content and deepfakes.

AI system and alleged parties

• AI system: unspecified
• AI purpose: Content Recommendation; Voice Generation
• Behaviour type: Autonomous
• Alleged developer: TikTok
• Alleged deployer: Unknown TikTok users from Vietnam, Unknown TikTok users from Nigeria, Unknown TikTok users from Iran, Unknown TiKTok users from China
• Alleged harmed parties: General public, Electoral integrity, Democracy, American electorate

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Minor, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Severe
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the report mentions that some accounts exploited TikTok's creator rewards program, potentially earning around $10,000 in payouts.

Directly caused: The creators of the Vietnamese network potentially obtained $10,000 from TikTok's creator rewards program through deceptive views.

Inferred additional harm: It is likely that other networks also extracted financial payouts from TikTok or other platforms, potentially totaling tens of thousands of dollars.

Malicious content

Reported: Yes, the report describes thousands of videos containing political lies, hyperbole, and divisive narratives spread by foreign networks.

Directly caused: The 91 identified accounts posted over 3,000 new videos in one month, pushing false and sometimes incendiary claims about political figures.

Inferred additional harm: It is highly likely that similar undetected networks are spreading additional toxic and malicious political content across TikTok and other platforms.

Democracy

Reported: Yes, the report explicitly notes that these foreign networks aim to cause chaos, erode trust in institutions, and corrode political discourse ahead of the 2024 presidential election.

Directly caused: The coordinated campaign targeted the 2024 U.S. presidential election with divisive narratives and falsehoods about candidates.

Indirectly caused: The proliferation of such content erodes public trust in democratic institutions and the integrity of the electoral process.

Inferred additional harm: The cumulative effect of these campaigns likely weakens democratic engagement and polarization among millions of voters.

Epistemic

Reported: Yes, the report describes the widespread dissemination of AI-generated political lies and fabricated narratives.

Directly caused: The accounts spread false stories, such as fake assassination videos and fabricated claims using AI-generated voices repeating identical scripts.

Indirectly caused: The spread of these falsehoods pollutes the information ecosystem, making it harder for users to distinguish real news from fake news.

Inferred additional harm: Millions of users who viewed and liked these videos may have developed false beliefs regarding political candidates and events.

People affected

• Occurrences reported: 1
• People reportedly exposed: 2500000

Potential causes

Management

• Creator Rewards Exploitation: Monetization policies inadvertently incentivize spam and fake content.
• Prioritizing Engagement Over Safety: Platform systems focus on user retention rather than content accuracy.
• Inadequate Proactive Risk Mitigation: Platform relies heavily on user flags rather than proactive detection.

Technology

• Low-Cost AI Voice Generation: AI tools allow cheap and easy generation of realistic voice scripts.
• Addictive Engagement Algorithms: Algorithms prioritize engagement, amplifying sensational fake news rapidly.
• Immature AI Detection Technology: Detection tech lags behind creation tech, making moderation difficult.

Data Inputs

• Stolen and Remixed Content: Accounts harvest and remix partisan commentary from other platforms.
• Lack of Content Labels: Inadequate labeling of AI-generated content hinders user awareness.
• Obscured Account Metadata: Fake profiles use deceptive locations and names to bypass regional checks.

Human Factors

• User Susceptibility to Clickbait: Users engage with sensationalized political lies, feeding the algorithm.
• Financial Profit Incentives: Creators exploit rewards programs to generate income from views.
• Lack of User Verification: Inability of users to verify the authenticity of viral political videos.

Process and Methods

• Evolving Evasion Tactics: Bad actors constantly change tactics to bypass platform defenses.
• Inadequate Account Moderation: Deleted accounts are instantly replaced by identical clones.
• Delayed AI Labeling Process: Automatic labeling of AI content was implemented too late.

Regulatory Environment

• Lack of Platform Accountability: Absence of strict regulations governing foreign misinformation campaigns.

Information quality

• Classification confidence: High
• Reason for confidence: The Wall Street Journal report provides a detailed, empirical investigation using automated bots and algorithmic analysis to identify and verify the 91 accounts. The findings are backed by TikTok's own internal investigation confirming the origins of the accounts.
• Ambiguities identified: The exact extent of state-backing remains unconfirmed, and the precise financial earnings of the networks are estimated rather than fully audited.
• Alternative interpretations: Some might argue the primary issue is human spam rather than an AI safety failure, though the use of AI generation and algorithmic recommendation is central to the scale of the impact.

Foreign networks from China, Nigeria, Iran, and Vietnam used AI-generated voices and scripts to scale disinformation targeting the 2024 U.S. presidential election. By exploiting TikTok's recommendation algorithm, these operations achieved millions of views and engagements, demonstrating an evolved information warfare capability aimed at eroding trust in democratic processes.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Foreign networks in China, Nigeria, Iran, and Vietnam

Threat characteristics

• Imminence: Near-term. Active campaign targeting an upcoming election, requiring ongoing monitoring and mitigation ahead of the vote.
• Autonomy: Human-supervised. Humans generated and deployed the campaigns, utilizing automated AI voice generators and TikTok's autonomous recommendation algorithm.
• Novelty: Evolved capability. Represents an evolution of existing foreign influence operations by integrating consumer-grade AI voice generation to scale content production.

Impact by dimension

• Physical security: Negligible. No physical infrastructure, kinetic weapons, or human safety threats were reported in this incident.
• Information security: Substantial. Coordinated foreign networks used AI-generated voices and cloned scripts to conduct election disinformation campaigns, accumulating millions of engagements.
• Sovereignty: Substantial. Foreign-linked networks deployed AI-generated content targeting the 2024 U.S. presidential election, aiming to erode trust in democratic processes.
• Economic security: Negligible. Financial impact was limited to minor exploitation of TikTok's creator rewards program, with no threat to strategic industries or economic stability.
• Societal stability: Substantial. The campaigns targeted millions of users with divisive narratives, aiming to cause political chaos and erode trust in democratic institutions.