Fake Accounts Using GAN Faces Deployed by Propaganda Campaign on Social Platforms

A pro-China propaganda campaign deployed fake accounts on Facebook, Twitter, and YouTube using GAN-synthesized faces to share and post comments on its content to gain wider circulation.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The campaign represents a coordinated effort by a foreign influence network to use AI-generated assets to conduct a political disinformation campaign targeting the U.S. public.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human threat actors who intentionally deployed GAN-generated profile pictures and automated voice-overs to run a coordinated propaganda campaign.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The incident involves the creation of AI-generated content, specifically synthetic profile pictures ('generative adversarial network' or GAN) and automated voice-overs, which falls under the category of AI-generated content such as deepfakes requiring transparency.

AI system and alleged parties

• AI system: generative adversarial network
• AI purpose: Deepfake Image Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: unknown
• Alleged deployer: Spamouflage Dragon
• Alleged harmed parties: YouTube users, Twitter Users, Facebook users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Minor
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the creation and dissemination of malicious propaganda videos and fake accounts using AI-generated profile pictures.

Directly caused: The campaign created fake social media accounts on Facebook, Twitter, and YouTube using GAN-generated profile pictures to mimic real users.

Indirectly caused: The fake accounts were used to share and post comments on English-language videos critical of U.S. policy and the Trump administration.

Inferred additional harm: It is likely that similar campaigns continue to generate synthetic profile pictures and automated voice-overs to populate influence networks, though their reach remains limited in this specific instance.

Democracy

Reported: The report explicitly describes an attempt to influence U.S. public opinion and sway political processes, though it was largely ineffective.

Directly caused: The campaign used AI-generated profile pictures and automated voice-overs to spread political propaganda critical of the U.S. administration's policies.

Indirectly caused: The activity aligns with warnings from U.S. intelligence officials that foreign governments attempt to sway public opinion to influence presidential elections.

Inferred additional harm: While this specific campaign failed to gain traction, the proliferation of such AI-enabled tactics poses a long-term risk of eroding trust in democratic processes and public discourse.

Epistemic

Reported: The report explicitly describes the creation of fake personas and automated voice-overs to spread misleading political narratives.

Directly caused: The campaign used GANs to generate fake human faces for profile pictures, creating the illusion of real users supporting the propaganda.

Indirectly caused: The fake accounts shared clumsy videos with automated voice-overs criticizing U.S. policies, attempting to distort the online information environment.

Inferred additional harm: The ease of generating realistic fake profile pictures makes it harder for users to distinguish between real and synthetic accounts, potentially degrading the overall epistemic quality of social media platforms.

People affected

• Occurrences reported: 1

Potential causes

Management

• Delayed Platform Detection: Social media platforms only reacted after the campaign was active.

Technology

• Publicly Available GANs: Allowed actors to easily generate fake faces to bypass reverse image searches.
• Flawed GAN Generation: Produced artifacts like matched eyeball positions and blurred backgrounds.
• Awkward Text-to-Speech: Generated unnatural voice-overs mispronouncing words like 'US' as 'us'.

Data Inputs

• Improper Training Data: AI struggled to render earrings due to limitations in facial training data.

Human Factors

• Clumsy Campaign Execution: Creators made obvious language errors and poorly configured automated tools.
• Threat Actor Exploitation: Foreign threat actors weaponized AI to scale up social media disinformation.

Process and Methods

• Poor Quality Control: Campaign released content with glaring linguistic and audio flaws.
• Weak Account Verification: Platforms lacked robust checks to detect newly generated synthetic profiles.

Regulatory Environment

• Lack of AI Regulations: No legal frameworks governed the malicious use of synthetic profile media.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from PCMag and Graphika provide clear, consistent, and detailed accounts of the 'Spamouflage Dragon' campaign, its use of GAN-generated profile pictures, and its lack of real-user engagement. The role of the AI is explicitly identified, and there are no conflicting accounts of the events.
• Ambiguities identified: The specific GAN models or services used by the actors are not named, and the exact origin of the campaign (whether state-sponsored) remains unconfirmed.
• Alternative interpretations: None. The event is clearly a coordinated inauthentic behavior campaign utilizing basic AI generation tools.

A pro-China propaganda campaign, Spamouflage Dragon, utilized GAN-generated profile pictures and automated text-to-speech to target U.S. politics. The operation had negligible national security impact due to poor execution and zero engagement from real users, but it highlights the evolving use of AI tools by foreign actors to conduct information warfare.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Spamouflage Dragon

Threat characteristics

• Imminence: Long-term. Reflects an ongoing strategic concern regarding foreign influence operations rather than an active crisis requiring immediate intervention.
• Autonomy: Human-controlled. Human operators created and deployed the accounts, utilizing GANs and text-to-speech as basic tools to generate content.
• Novelty: Evolved capability. Represents an early evolution of foreign influence operations incorporating AI-generated synthetic faces and automated audio.

Impact by dimension

• Physical security: Negligible. No physical infrastructure, kinetic targeting, or human safety concerns were involved in this digital propaganda campaign.
• Information security: Minor. Represented an attempt by a pro-China network to conduct a coordinated disinformation campaign targeting U.S. policy, but it had negligible real-world impact due to poor quality and zero authentic user engagement.
• Sovereignty: Minor. Attempted to influence domestic U.S. political discourse and criticize the administration, representing a foreign influence effort, though it failed to disrupt any government functions.
• Economic security: Negligible. No financial systems, strategic technologies, or economic supply chains were targeted or impacted.
• Societal stability: Negligible. The campaign failed to gain traction or engagement, resulting in no measurable impact on social cohesion or human rights.