Scammers Using Deepfakes of Women's Faces and Voices for False and Offensive Advertisements

Scammers used AI tools from HeyGen and ElevenLabs to create deepfake videos of influencers Michel Janse, Olga Loiek, Shadé Zahrai, and Carrie Williams, misusing Lana Smalls's voice in Williams's case. These videos promoted offensive products and false messages, in some cases targeting nationalist Chinese men to boost China-Russia ties, causing emotional distress and damaging the victims' reputations.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The primary harm involves scammers using AI tools to clone individuals' likenesses for unauthorized commercial endorsements, scams, and targeted sexualized humiliation.

Additional risk subdomains

• 4.1 Disinformation, surveillance, and influence at scale: The deepfakes were also used in coordinated campaigns on Chinese social media to spread pro-Russia political propaganda and stoke nationalistic sentiment.
• 1.2 Exposure to toxic content: The deepfakes exposed users to highly inappropriate and offensive content, including sexualized ads referencing adult film audio and erectile dysfunction.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The risk is driven by human scammers who intentionally deploy and use AI cloning tools to create unauthorized deepfakes for financial and political gain.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes AI-generated content such as deepfakes, which pose a moderate risk and require specific transparency obligations to ensure users are informed of the AI's nature.

AI system and alleged parties

• AI system: Eleven Labs, HeyGen (ElevenLabs, HeyGen)
• AI purpose: Deepfake Video Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: HeyGen, ElevenLabs
• Alleged deployer: Unknown scammers
• Alleged harmed parties: Shadé Zahrai, Olga Loiek, Michel Janse, Lana Smalls, Carrie Williams

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Minor, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report explicitly describes the creation and spread of toxic and malicious deepfake content.

Directly caused: Scammers created and spread deepfake videos containing highly inappropriate sexualized content and political propaganda across platforms like YouTube and Douyin.

Inferred additional harm: Given the accessibility of these AI tools, it is highly likely that thousands of other toxic and non-consensual deepfake videos are being created and spread.

Differential treatment

Reported: Yes, the report notes that this technology disproportionately targets women, representing a form of nationalistic sexism and sexual objectification.

Directly caused: Female influencers and private individuals were targeted for sexualized objectification and nationalistic propaganda, exploiting their gender and likenesses.

Inferred additional harm: Women are likely to continue being disproportionately targeted by deepfake scams and sexualized content, leading to systemic unequal safety outcomes online.

Civil rights

Reported: Yes, the report describes violations of the victims' civil rights regarding the unauthorized use of their likenesses and identities.

Directly caused: The unauthorized cloning of the victims' voices and faces violated their personal autonomy and right to control their own likenesses.

Inferred additional harm: As deepfake technology proliferates without adequate legal frameworks, widespread violations of individual privacy and identity rights are likely to continue unchecked.

Democracy

Reported: Yes, the report describes how deepfakes are used to stoke patriotic fervor, spread pro-Russia propaganda, and influence public opinion on Chinese social media.

Directly caused: AI-manipulated videos of Ukrainian and other foreign women were used to spread political disinformation and stoke nationalistic sentiment in China.

Inferred additional harm: The widespread use of deepfakes for political propaganda and disinformation campaigns threatens to erode public trust in media and democratic discourse globally.

Psychological

Reported: Yes, the report explicitly describes psychological distress and feelings of violation among the victims.

Directly caused: Victims like Loiek, Smalls, and Janse experienced severe emotional distress, feeling violated and mortified by the unauthorized sexualized or political use of their likenesses.

Inferred additional harm: It is highly likely that other cloned individuals suffered similar psychological distress and anxiety upon discovering their deepfakes.

Epistemic

Reported: Yes, the report explicitly describes epistemic harm through the fabrication of statements and identities.

Directly caused: The deepfakes fabricated statements, making victims appear to say things they never said (e.g., praising Putin, discussing erectile dysfunction), misleading viewers.

Inferred additional harm: The proliferation of highly realistic deepfakes erodes the shared consensus reality, making it difficult for the public to distinguish between real and synthetic media.

People affected

• Occurrences reported: 4
• People reportedly harmed: 4
• People reportedly exposed: 360000

Potential causes

Management

• Delayed Safety Policy Deployment: YouTube delayed executing promised policies to allow synthetic content removal.
• Profit Over Safety Focus: Scammers exploit automated ad systems that prioritize volume over verification.
• Lack of Proactive Monitoring: Platform management failed to proactively audit advertiser accounts.

Technology

• Advanced Voice and Video Cloning: Tools from HeyGen and Eleven Labs allow realistic voice and lip-sync cloning.
• Low Cost of AI Generation: Cheap promotional tools drastically lower the cost for scammers to run ads.
• Lack of Robust Deepfake Detection: Platforms struggle to automatically detect and flag realistic AI deepfakes.

Data Inputs

• Publicly Available Social Media Media: A few seconds of public video/audio are enough to clone a person's likeness.
• Cross-Platform Video Scraping: Scammers scrape content from Western social media to use on Chinese platforms.
• Lack of Source Verification Data: Platforms lack metadata to verify if uploaded videos contain stolen likenesses.

Human Factors

• User Naivety to Deepfake Risks: Victims believed deepfakes only targeted famous political or public figures.
• Financial and Nationalist Motives: Scammers leverage nationalistic sentiment to drive traffic and sales.
• Masculine and Nationalistic Pride: Target audience consumes objectifying content to reaffirm pride.

Process and Methods

• Slow Platform Takedown Processes: YouTube and other platforms were slow to implement deepfake removal policies.
• Evasion of Automated Detectors: Scammers bypass landing page detectors by hosting clean redirect pages.
• Inadequate Cross-Border Monitoring: Closed internet environments prevent foreign victims from finding stolen content.

Regulatory Environment

• Absence of Federal Deepfake Laws: No federal laws exist to govern deepfakes outside of politics and porn.
• Jurisdictional Enforcement Gaps: Global scammers exploit jurisdictional boundaries to avoid legal prosecution.
• Costly Legal Recourse for Victims: Bringing civil cases for likeness theft is too expensive for normal victims.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, detailed, and consistent accounts of multiple individuals whose likenesses were stolen and cloned using AI tools. The roles of HeyGen and ElevenLabs are explicitly mentioned, and the real-world harms (emotional distress, non-consensual commercial/political exploitation) are well-documented with direct quotes from the victims.
• Ambiguities identified: The exact identity of the scammers and the full scale of the operations remain partially anonymous, though the mechanisms of the AI tools used are clear.
• Alternative interpretations: None. The incidents are clearly cases of unauthorized deepfake creation for commercial gain and political propaganda.

Scammers used consumer AI tools from HeyGen and ElevenLabs to clone the likenesses of individuals for unauthorized commercial advertisements and localized pro-Russia political propaganda. While violating individual privacy and contributing to online disinformation, the incident represents a minor national security threat managed through standard platform moderation and legal frameworks.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, Ukraine, Australia, China
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding the proliferation of synthetic media and identity theft rather than an immediate crisis.
• Autonomy: Human-controlled. The AI tools are used as software by human scammers who manually input source videos and deploy the outputs.
• Novelty: Evolved capability. While deepfakes are an established threat, the ease of creation and cross-lingual translation represents an evolved and highly accessible capability.

Impact by dimension

• Physical security: Negligible. No physical infrastructure, kinetic weapons, or human safety systems were targeted or affected in this incident.
• Information security: Minor. Deepfakes were used to spread pro-Russia political propaganda on Chinese social media using cloned likenesses of foreign individuals, representing a minor information operations impact.
• Sovereignty: Negligible. No core government operations, electoral systems, or sovereignty decisions were compromised or targeted.
• Economic security: Minor. Minor economic impact restricted to individual commercial scams and potential reputational damage to content creators, with no systemic threat to financial systems.
• Societal stability: Minor. The unauthorized cloning of individuals' likenesses violates personal privacy and civil rights, but does not threaten large-scale societal stability.