China Targets AI-Driven Fraud and Deepfake Scandals with New Crackdowns

Chinese law enforcement has targeted a rise in AI-driven crimes. The crimes include deepfake and voice synthesis used for fraud, identity theft, and unauthorized personality rights usage. In particular, "AI undressing" scams, fake relationships using synthesized voices, and game hacking software make up many of these cases. In response, authorities have prosecuted multiple cases and implemented stricter regulations to control AI misuse.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident report describes multiple instances of criminals using generative AI for fraud, scams, game cheating, and creating non-consensual sexual imagery to gain financial advantage.

Additional risk subdomains

• 3.1 False or misleading information: Criminals used generative AI to automatically generate scripts and synthesize fake videos, such as a fabricated industrial park fire, to gain traffic.
• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: A suspect used AI-synthesized dynamic face models to bypass face-recognition systems and illegally download a company's electronic business license.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The harms were caused by human criminals intentionally misusing deployed generative AI systems to commit fraud, identity theft, and generate non-consensual pornographic images.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the widespread use of generative AI to create deepfakes, synthetic voices, and manipulated videos, which fall under AI-generated content requiring transparency obligations.

AI system and alleged parties

• AI system: generative artificial intelligence service
• AI purpose: Deepfake Image Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: Unknown voice synthesis technology developers, Unknown game cheating technology developers, Unknown deepfake technology developers
• Alleged deployer: Zeng Moumou, Wang Mouhe, Unknown scammers, Unknown deepfake creators, Tang Mou, Bai Moumou, AI fraud rings in China
• Alleged harmed parties: Chinese general public, Chinese citizens

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Substantial, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the report explicitly describes financial losses and illegal gains across multiple cases.

Directly caused: Wang made 6.29 million RMB from AI game hacks; Bai made over 10,000 RMB from deepfakes; Zeng made 90,000 RMB from identity theft; the extortion gang stole over 1 million RMB from 50+ victims; and the voice-cloning court case ordered 250,000 RMB in compensation.

Inferred additional harm: The 79 cracked AI face-swap fraud cases likely resulted in millions of RMB in additional unquantified financial losses for the victims.

Malicious content

Reported: Yes, the report explicitly describes the creation and sale of deepfake pornographic images.

Directly caused: The suspect Bai generated and sold nearly 7,000 deepfake pornographic images using 'one-click undressing' AI technology.

Privacy

Reported: Yes, the report explicitly describes privacy and personal information violations.

Directly caused: Suspect Zeng used stolen names, ID numbers, and photos to synthesize dynamic face models of over 1,000 people to bypass app security and download a company's electronic business license.

Inferred additional harm: The 79 face-swap fraud cases and the 'one-click undressing' deepfakes suggest widespread unauthorized acquisition and processing of private personal images.

Epistemic

Reported: Yes, the report describes the creation and dissemination of fake videos and rumors.

Directly caused: A criminal group used generative AI to automatically generate scripts and synthesize over 3,000 fake videos, including a fabricated industrial park fire, to gain online traffic.

Inferred additional harm: The widespread dissemination of thousands of AI-generated fake videos likely misled online users and degraded the local information ecosystem.

People affected

• Occurrences reported: 85
• People reportedly harmed: 131
• People reportedly exposed: 1100

Potential causes

Management

• Inadequate Risk Assessments: App operators failed to assess risks of AI-driven authentication bypass.
• Weak Data Security Management: Failure of entities to secure ID photos, leading to massive leaks.
• Platform Self-Regulation Failures: Service providers failed to monitor and block malicious AI tool generation.

Technology

• Low AI Standardization: Low standardization and high customization make AI easy to abuse.
• Flawed Liveness Detection: Vulnerabilities in face recognition allowed dynamic AI face-swaps to bypass.
• Simple AI Video Synthesis: Generative AI tools make it too easy to synthesize realistic fake videos.

Data Inputs

• Leaked Identity Photos: Stolen ID photos and personal info used to generate deepfake face matches.
• Unauthorised Voice Data: Unauthorised use of voice recordings to train text-to-speech AI models.
• Scraped Video Fragments: Scraped online videos used as raw inputs to synthesize fake disaster clips.

Human Factors

• Financial Motivation of Criminals: Criminals are driven by easy profits from selling deepfakes and AI cheats.
• Weak Personal Privacy Awareness: Users exposing sensitive biometric data online, easing identity theft.
• Desire for Web Traffic: Content creators fabricating fake disaster videos to gain online traction.

Process and Methods

• Weak App Verification Processes: Government and payment apps failed to detect synthetic dynamic face inputs.
• Inadequate Content Filtering: Platforms lack robust mechanisms to filter AI-synthesized fake news.
• Easy AI Tool Accessibility: Lack of strict access controls on powerful AI deepfake and cheating tools.

Regulatory Environment

• Lagging Legal Frameworks: Existing laws struggle to keep pace with rapid AI crime evolution.
• Lack of Strict Platform Oversight: Insufficient oversight on platforms hosting illegal AI services and tools.
• Complex Evidence Collection: High anonymity and complexity of AI crimes hinder law enforcement efforts.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are official Chinese state media and police releases detailing specific criminal cases, arrests, and court rulings with precise figures (e.g., 7,000 deepfakes, 6.29 million RMB, 50+ victims, 250,000 RMB compensation). This provides a highly reliable factual basis for classification.
• Ambiguities identified: The specific AI models and software used by the criminals are not named, referred to generally as 'generative AI' or 'AI software'.
• Alternative interpretations: None. The cases are clearly defined as criminal misuses of AI technology.

The incident details a surge in domestic cyber-crime in China leveraging generative AI for identity theft, deepfake fraud, and bypassing government face-recognition systems. While these activities represent an evolved and scaled threat to individual privacy and administrative security, their impact is currently manageable within standard law enforcement procedures and poses minor national security risk.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: China
• Alleged perpetrator: Domestic criminals

Threat characteristics

• Imminence: Long-term. Represents an ongoing, strategic cyber-criminal capability trend rather than an active, imminent national security crisis.
• Autonomy: Human-controlled. AI tools like generators, face-swappers, and voice-cloners were directly operated and controlled by human criminals to execute specific fraudulent tasks.
• Novelty: Evolved capability. Represents a significant advancement and scaling of existing fraud and identity theft methods using easily accessible generative AI tools.

Impact by dimension

• Physical security: Negligible. No physical systems, kinetic targets, or critical infrastructure control systems were impacted or manipulated.
• Information security: Minor. Generative AI was used to create fake videos and deepfakes, but these were localized criminal fraud and traffic-generation schemes rather than coordinated state-level information warfare.
• Sovereignty: Minor. A criminal bypassed government app face-recognition security to download electronic business licenses, representing a minor breach of administrative verification systems.
• Economic security: Minor. The incidents caused millions of RMB in localized financial losses due to fraud and game hacking, but they do not threaten strategic industries or national economic stability.
• Societal stability: Minor. Widespread misuse of biometric data and deepfake technology compromised individual privacy and civil rights, but the issues remain manageable within standard law enforcement procedures.