Amandine Le Pen Deepfake Account Misleads Thousands on TikTok

A TikTok account, "Amandine Le Pen," created using AI deepfake technology, impersonated a fictional niece of Marine Le Pen, amassing over 30,000 followers. The account spread pro-RN messages and solicited donations, misleading users and exploiting political influence. Visual inconsistencies revealed the deepfake, raising concerns about AI misuse for political manipulation, identity theft, and violation of personal rights, especially with similar accounts proliferating, such as "Lena Maréchal Lepen."

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The deepfake accounts were used to conduct a coordinated campaign aimed at influencing public opinion and political processes ahead of the EU elections.

Additional risk subdomains

• 4.3 Fraud, scams, and targeted manipulation: The accounts used fake personas to solicit financial donations from followers via virtual coffee purchases.
• 1.2 Exposure to toxic content: The deepfake videos featured sexually suggestive content and xenophobic comments mocking people of color.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human creators who intentionally generated and deployed deepfake videos of fictional Le Pen family members to manipulate public opinion and solicit donations.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The EU AI Act classifies AI-generated content such as deepfakes under Limited Risk, which imposes specific transparency obligations to ensure users are informed that they are interacting with AI-generated content.

AI system and alleged parties

• AI system: unspecified
• AI purpose: Deepfake Video Generation; Social Media Content Generation
• Behaviour type: Tool
• Alleged developer: Unknown deepfake creators
• Alleged deployer: Unknown TikTok user
• Alleged harmed parties: Le Pen family, French general public

Harm severity

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Minor, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report describes deepfake videos of the fictional nieces dancing in their underwear and mocking people of color.

Directly caused: The creators generated and spread deepfake videos featuring sexually suggestive imagery and xenophobic comments mocking people of color.

Indirectly caused: N/A

Inferred additional harm: The spread of these videos on TikTok likely contributed to online harassment and the normalization of xenophobic discourse among viewers.

Differential treatment

Reported: Yes, the report notes that the deepfake videos mocked people of color.

Directly caused: The AI-generated content targeted and mocked people of color, promoting discriminatory far-right rhetoric.

Indirectly caused: N/A

Inferred additional harm: N/A

Civil rights

Reported: Yes, the report mentions potential violations of personality rights and the right to one's image.

Directly caused: The unauthorized use of Marine Le Pen's and Marion Marechal's likenesses, as well as the stolen videos of real influencers, violated their personality and image rights.

Indirectly caused: N/A

Inferred additional harm: N/A

Democracy

Reported: Yes, the report highlights concerns about political manipulation and the spread of misleading content ahead of the EU elections.

Directly caused: The deepfake accounts were used to spread political propaganda and manipulate public opinion ahead of the June EU elections.

Indirectly caused: N/A

Inferred additional harm: The proliferation of such deceptive political deepfakes risks undermining public trust in democratic electoral processes and political figures.

Privacy

Reported: Yes, the report mentions the unauthorized use of images and videos stolen from real female influencers.

Directly caused: The creators stole videos of real female influencers and mapped deepfake faces onto them without consent.

Indirectly caused: N/A

Inferred additional harm: N/A

Epistemic

Reported: Yes, the report describes the creation of fictional personas presented as real members of the Le Pen family.

Directly caused: The campaign fabricated non-existent Le Pen family members to spread political messages, misleading thousands of followers who believed they were real.

Indirectly caused: N/A

Inferred additional harm: The widespread dissemination of highly realistic deepfakes contributes to the erosion of shared reality and makes it harder for the public to distinguish real political figures from synthetic ones.

People affected

• Occurrences reported: 1
• People reportedly harmed: 5
• People reportedly exposed: 2000000

Potential causes

Management

• Poor Platform Oversight: TikTok management did not prioritize election-related deepfake risks.

Technology

• Generative AI Face Editing: AI tools allowed editing faces onto stolen influencer videos easily.
• Weak Deepfake Detection: TikTok algorithms failed to detect facial and mouth movement anomalies.

Data Inputs

• Stolen Influencer Videos: Real videos of influencers were used without consent as base inputs.
• Unlabeled AI Content: Synthesized videos lacked metadata or labels indicating AI generation.

Human Factors

• Malicious AI Exploitation: Anonymous users created fake personas to manipulate political views.
• User Gullibility: Viewers believed the fake Le Pen family members were real people.

Process and Methods

• Slow Content Moderation: TikTok failed to remove or label the viral deepfakes in a timely manner.
• Lack of Proactive Screening: Platforms relied on manual reports rather than automatic detection.

Regulatory Environment

• Delayed DSA Implementation: France lagged in designating trusted flaggers and coordinating bodies.
• Weak Legal Protections: MPs voted against banning deepfakes published without consent.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from RTS and Euractiv provide consistent, detailed accounts of the deepfake accounts, their reach, the specific content of the videos, and the political context. The facts regarding the creation of fictional Le Pen family members and the use of stolen influencer videos are clearly established.
• Ambiguities identified: The exact identity of the creators remains unconfirmed, with one operator claiming it was a 'social experiment' while others suggest financial or political motives.
• Alternative interpretations: The campaign could be interpreted purely as a financial scam rather than a coordinated political disinformation campaign, though the heavy use of political hashtags and election references strongly supports the political interpretation.

This incident involved a coordinated political disinformation campaign in France using AI-generated deepfakes of fictional relatives of politician Marine Le Pen. The accounts amassed millions of views, spreading xenophobic content and far-right propaganda ahead of the EU elections, highlighting the evolving threat of synthetic media in democratic processes.

• Overall national security impact: Substantial
• Response level: Substantial
• Scope: Single nation
• Primary target: France
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Near-term. The threat was active and aimed at influencing upcoming EU elections, requiring rapid counter-disinformation response.
• Autonomy: Human-controlled. Creators manually generated the deepfakes and operated the TikTok accounts to post content and solicit funds.
• Novelty: Evolved capability. Represents an evolved use of deepfakes by fabricating entirely fictional political personas rather than just impersonating existing figures.

Impact by dimension

• Physical security: Negligible. No physical threat or critical infrastructure impact reported in connection with this social media deepfake campaign.
• Information security: Substantial. AI-generated deepfakes created fictional political family members to spread far-right propaganda and xenophobic content, reaching millions of views ahead of EU elections.
• Sovereignty: Minor. Attempted political manipulation ahead of EU elections, though quickly mitigated by media exposure and platform account deletion.
• Economic security: Negligible. Minor financial solicitation via virtual donations, with negligible impact on national economic or technological security.
• Societal stability: Minor. Promoted xenophobic rhetoric and violated personality rights of influencers and politicians, but lacked scale to trigger widespread civil unrest.