Purportedly AI-Manipulated Synthetic Media Allegedly Fuels False Reports of Natural Disasters and Accidents in China

In China, purported AI tools were reportedly used to fabricate and disseminate false reports of disasters, including a landslide in Yunnan, an earthquake in Sichuan, and a sudden death after a traffic incident. On May 27, 2024, a real 5.0-magnitude earthquake occurred in Muli County, Sichuan, with no casualties and limited property damage. However, a social media post later falsely claimed the epicenter was in Xide County and exaggerated the event's severity, adding fabricated images of extensive destruction. The deployers of these false reports have since received administrative penalties from Chinese authorities for their actions.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The perpetrators intentionally used AI tools to fabricate false news stories and images to manipulate public beliefs, attract traffic, and gain financial platform rewards.

Additional risk subdomains

• 3.1 False or misleading information: The AI tools were used to generate highly realistic but entirely fabricated news articles and images regarding disasters and public safety incidents.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The risk was caused by human actors who intentionally used deployed AI tools to generate and spread false news stories for personal gain.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: The report describes the use of AI to generate synthetic text and images (deepfakes/synthetic media) to fabricate news, which falls under Limited Risk due to transparency and disclosure obligations for AI-generated content.

AI system and alleged parties

• AI system: AI software
• AI purpose: Social Media Content Generation; Deepfake Image Generation
• Behaviour type: Tool
• Alleged developer: Unknown deepfake technology developers, Unknown AI developers
• Alleged deployer: Yang Moumou, Tian Mou, Lou Moumou
• Alleged harmed parties: Yunnan general public, Xide County in Sichuan residents, Sichuan general public, Muli County in Sichuan residents, Chinese general public, Chinese citizens, Chengcheng County Shaanxi residents

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report explicitly describes the creation and dissemination of false rumors and fabricated news stories using AI tools.

Directly caused: Perpetrators used AI to generate fake news articles and images regarding a landslide, an earthquake, and a police enforcement death, which were published online.

Inferred additional harm: It is likely that these fabricated stories reached thousands of social media users, exposing them to deceptive and potentially distressing content.

Epistemic

Reported: Yes, the report explicitly describes the generation of false information and fabricated news stories that misled the public.

Directly caused: AI tools were used to generate fake articles and images, such as a fabricated death toll for a landslide and a fake story about a woman dying during police enforcement.

Inferred additional harm: The ease of generating realistic fake news with AI likely contributes to a broader erosion of shared truth and public trust in online information.

People affected

• Occurrences reported: 3

Potential causes

Management

• Incentivizing Traffic over Truth: Platforms reward high-traffic posts, encouraging sensational AI rumors.
• Lack of Source Verification: Platforms host unverified AI stories without checking their authenticity.

Technology

• AI Generation Tools Accessibility: AI tools make it easy to synthesize realistic fake text and images.
• Realistic Image Synthesis: AI software generates highly convincing images of non-existent disasters.

Data Inputs

• Unverified Internet Sources: Users feed unverified online rumors into AI software to expand them.
• Grafting AI Text with Real Images: Combining AI-generated text with actual photos to mislead readers.

Human Factors

• Desire for Traffic and Profit: Perpetrators fabricate rumors to gain attention and platform rewards.
• Deliberate Malicious Intent: Users intentionally use AI to generate sensationalized fake news.

Process and Methods

• Automated Content Creation: Using AI to automatically write full articles without manual verification.
• Inadequate Platform Moderation: Social media platforms fail to block AI-generated rumors immediately.

Regulatory Environment

• Reliance on Post-Hoc Punishment: Law enforcement acts after the rumors have already spread and caused harm.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly details three specific cases, the names of the perpetrators, the dates, the specific lies fabricated, and the legal outcomes. The details provided are sufficient to confidently classify the risk domains and harms.
• Ambiguities identified: The specific AI software or models used by the perpetrators are not identified.

Domestic individuals in China used AI text and image generation tools to fabricate and spread false news reports about natural disasters and police misconduct to gain social media traffic and financial rewards. The incident was managed via standard local administrative penalties and had minor national security implications, primarily affecting local social order and public trust in official communications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: China
• Alleged perpetrator: Domestic individuals (Yang, Luo, and Tian)

Threat characteristics

• Imminence: Long-term. The events occurred in early-to-mid 2024 and have already been resolved with administrative penalties, representing an ongoing strategic concern rather than an active crisis.
• Autonomy: Human-controlled. Human perpetrators intentionally used AI tools to generate text and images; the AI did not act autonomously.
• Novelty: Established threat. Spreading fake news and rumors online using text and image generators is a well-established threat that has occurred frequently.

Impact by dimension

• Physical security: Negligible. The incident involved online rumors about natural disasters and police enforcement, but did not cause physical damage, kinetic attacks, or disruption to critical infrastructure.
• Information security: Minor. AI-generated misinformation was spread online, but it was done by domestic individuals for personal financial gain, not as part of a coordinated foreign state-sponsored information warfare campaign.
• Sovereignty: Minor. The fake stories targeted local government actions (police enforcement, disaster response), but the impact was limited, managed via local administrative penalties, and did not threaten state sovereignty.
• Economic security: Negligible. The perpetrators sought minor platform rewards. There was no strategic technology theft, financial system attack, or loss of technological competitive advantage.
• Societal stability: Minor. Fabricated news caused local social disruption and negative social impact, but did not lead to large-scale civil unrest, systemic human rights violations, or population-scale manipulation.