AI Video-Making Tool Abused to Deploy Pro-China News on Social Media

Synthesia's AI-generated video-making tool was reportedly used by Spamouflage to disseminate pro-China propaganda news on social media using videos featuring highly realistic fictitious news anchors.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The incident involved a state-aligned influence operation using AI-generated avatars to conduct a coordinated disinformation campaign aimed at manipulating public opinion and political processes.

Additional risk subdomains

• 7.3 Lack of capability or robustness: Synthesia's content moderation system failed to detect and block the policy-violating political content.
• 3.1 False or misleading information: The AI system was used to generate false personas and misleading news broadcasts to deceive viewers.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human actors who intentionally used a deployed commercial AI video generation platform to create deceptive political propaganda.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is used for generating AI-generated content such as deepfakes, which under the EU AI Act requires transparency obligations to ensure users are aware they are interacting with AI-generated media.

AI system and alleged parties

• AI system: Synthesia AI video creation platform (Synthesia)
• AI purpose: Deepfake Video Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: Synthesia
• Alleged deployer: Spamouflage Dragon
• Alleged harmed parties: YouTube users, Twitter Users, Synthesia, Facebook users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Minor, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report describes the creation of pro-Chinese propaganda videos featuring fictitious news anchors.

Directly caused: The AI platform was used to directly generate deceptive political videos criticizing the US government and promoting Chinese state interests.

Inferred additional harm: It is likely that other malicious actors have used or will use similar commercial AI tools to generate undetected political propaganda.

Democracy

Reported: Yes, the report highlights concerns that deepfake technology could corrode public trust in institutions and governments.

Directly caused: The state-aligned operation used deepfakes to spread political propaganda, aiming to influence public opinion and undermine democratic institutions.

Inferred additional harm: The proliferation of such state-aligned deepfake campaigns could lead to a broader erosion of public trust in democratic processes and media institutions.

Epistemic

Reported: Yes, the report describes how deepfake videos blur the line between fact and fiction, severing people's ability to discern reality.

Directly caused: The AI system generated highly realistic, fictitious news anchors to present fabricated news broadcasts, misleading viewers about the authenticity of the presenters.

Inferred additional harm: The widespread availability of commercial deepfake tools is likely to further degrade the shared information ecosystem and weaken consensus reality.

People affected

• Occurrences reported: 2
• People reportedly exposed: 600

Potential causes

Management

• Small Content Moderation Team: Synthesia only had a four-person team dedicated to preventing misuse.
• Reliance on Post-Hoc Bans: Company suspended accounts only after the deepfakes were exposed by others.
• Difficult Risk Assessment: Management found it hard to identify and block grey-area misinformation.

Technology

• Highly Realistic AI Avatars: Synthesia's AI creates lifelike digital twins that mimic real humans.
• Rapidly Evolving Deepfakes: Technology allows easy cloning of gestures, body language, and voices.
• Generative Adversarial Networks: GANs enable the creation of unique, realistic faces not in training data.

Data Inputs

• Text-to-Speech Scripts: Users typed scripts that the AI actors read out in multiple languages.
• Use of Hired Actor Likenesses: Avatars are based on real actors' appearances, making them highly realistic.
• Single Image Manipulation: AI can clone and manipulate faces based on just a single image input.

Human Factors

• High Public Trust in Videos: People instinctively trust video content more than static images online.
• Low Public Visual Literacy: General public lacks the skills to distinguish deepfakes from real videos.
• Malicious Actor Intent: State-aligned groups actively exploit AI tools to spread political propaganda.

Process and Methods

• Weak Moderation of Grey Areas: Moderation systems struggle to detect subtle political misinformation.
• Easy Commercial Access: Low-cost subscriptions allow anyone to generate AI videos in minutes.
• Ineffective Social Media Filters: Platforms like Twitter and Facebook failed to automatically block the videos.

Regulatory Environment

• Lack of Clear AI Laws: Absence of comprehensive regulations to manage the spread of deepfake tools.
• Lagging Regulatory Frameworks: Technological advances run far ahead of existing digital age legislation.
• No Mandatory Disclosure Rules: Lack of legal requirements to label AI-generated content as synthetic.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent details about the nature of the AI system, its developer (Synthesia), the actors involved (Spamouflage), and the specific videos produced. The technical mechanism of the AI (text-to-video avatar generation) and the failure of content moderation are explicitly described.
• Ambiguities identified: The exact number of views and total reach of the Burkina Faso campaign are not fully detailed.

A pro-China influence operation utilized commercial AI video generation software to create fake news anchors for political propaganda. While the immediate national security impact is minor due to low quality and very low engagement, the incident establishes a significant precedent as the first known state-aligned use of generative AI for political disinformation.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: United States
• Other affected: Burkina Faso
• Alleged perpetrator: China-aligned actor

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic capability and evolving disinformation technique rather than an active, immediate crisis.
• Autonomy: Human-controlled. The AI tool was directly controlled by human operators who wrote the scripts and initiated the video generation process.
• Novelty: First-of-its-kind. Marks the first documented instance of a state-aligned actor deploying AI-generated fictitious personas for political disinformation campaigns.

Impact by dimension

• Physical security: Negligible. The incident involved purely digital information operations with no physical threats, kinetic attacks, or critical infrastructure disruption.
• Information security: Minor. State-aligned actors utilized AI-generated deepfake news anchors to spread political propaganda. However, the operational impact was minor due to very low engagement and low video quality.
• Sovereignty: Negligible. While targeting U.S. political topics, the operation did not compromise government decision-making, electoral systems, or core state sovereignty.
• Economic security: Negligible. No critical economic infrastructure, financial systems, or strategic technologies were targeted or compromised during this incident.
• Societal stability: Negligible. The campaign attempted to influence public opinion but had negligible societal impact or threat to human rights due to extremely limited reach.