Chinese Businessman Reportedly Defrauded of 4.3 Million Yuan by AI-Generated Deepfake Impersonating Friend

A scammer in China used AI-generated deepfake technology to impersonate a businessman’s trusted friend in a video call, convincing him to transfer 4.3 million yuan ($612,000). The fraudster mimicked the friend’s face and voice, claiming another associate needed funds for a business transaction. Without verifying the transfer, the victim sent the money. He later realized the deception and contacted police, who managed to recover 3.4 million yuan.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The primary incident involves a scammer using AI deepfake technology to impersonate a trusted friend and defraud a businessman of 4.3 million yuan.

Additional risk subdomains

• 3.1 False or misleading information: The report also describes a man using ChatGPT to generate and spread a fake news article about a deadly bus crash.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incidents were caused by human actors intentionally misusing deployed AI technologies (deepfakes and ChatGPT) to commit financial fraud and spread disinformation.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the use of chatbots (ChatGPT) and AI-generated content such as deepfakes, which are classified as limited risk under the EU AI Act and subject to transparency obligations.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Deepfake Video Generation; Writing Assistant
• Behaviour type: Tool
• Alleged developer: Unknown voice cloning technology developers, Unknown deepfake technology developers
• Alleged deployer: Unknown scammers, Scammers impersonating friend of Guo
• Alleged harmed parties: Guo (surname of deepfake scam victim), General public of China, Friend of Guo (surname of deepfake scam victim)

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the report describes a financial loss of 4.3 million yuan (approximately S$823,000) from the deepfake scam, with 3.4 million yuan later recovered by police.

Directly caused: The average financial loss per occurrence for the scam was 4.3 million yuan (S$823,000), with a net unrecovered loss of 900,000 yuan (S$172,000) at the time of reporting.

Inferred additional harm: The victim may have incurred additional indirect financial costs, such as legal or administrative fees, to recover the frozen funds.

Malicious content

Reported: Yes, the report describes the creation of a fake news article about a deadly bus crash and the use of deepfake technology to impersonate a friend for fraud.

Directly caused: The scammer directly created a deepfake video and voice clone to perpetrate fraud. In the second incident, a man directly generated a fake news article about a deadly bus crash using ChatGPT.

Epistemic

Reported: Yes, the report describes the generation and wide dissemination of a fake news article about a deadly bus crash.

Directly caused: A man used ChatGPT to fabricate a fake news article about a fatal bus crash, which was spread widely on social media, misleading readers.

Inferred additional harm: The spread of fake news about public safety incidents can erode public trust in local news and official information channels.

People affected

• Occurrences reported: 2
• People reportedly harmed: 2
• People reportedly exposed: 1002

Potential causes

Management

• Inadequate Corporate Risk Controls: Company account lacked strict multi-authorization protocols for transfers.

Technology

• AI Face and Voice Cloning: Con artists used smart AI technology to change their face and voice.
• Generative AI Accessibility: Abuse of generative tools for creating highly convincing fake media.

Data Inputs

• Fraudulent Payment Records: Scammer sent a fake screenshot of a payment record to deceive the victim.
• Stolen Identity Data: Scammer used the friend's personal data to impersonate him convincingly.

Human Factors

• Overreliance on Visuals: Victim let down guard after verifying the face and voice on video call.
• Failure to Out-of-Band Verify: Victim did not check bank account or call friend before sending money.

Process and Methods

• Inadequate Transaction Verification: No robust process to verify large transfers before executing them.
• Lack of Deepfake Detection Tools: Absence of real-time deepfake detection during the video call.

Regulatory Environment

• Lagging Deepfake Regulations: Regulations on deepfakes and AI safety assessments are still emerging.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, factual details about the deepfake scam, including the specific amount of money stolen and recovered, the location, and the method of deception. It also clearly details the separate ChatGPT incident. There is little ambiguity regarding the role of AI in these events.
• Ambiguities identified: The exact technical deepfake software used by the scammer is not specified.

The incidents involve the criminal misuse of generative AI in China, specifically real-time deepfake technology to execute a 4.3 million yuan financial fraud and ChatGPT to generate fake news about a bus crash. While demonstrating the evolving sophistication of AI-enabled crime and localized disinformation, the events represent minor domestic law enforcement challenges rather than significant threats to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: China
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incidents represent ongoing criminal and disinformation capabilities that require regulatory and law enforcement monitoring, rather than an active national security crisis.
• Autonomy: Human-controlled. The AI systems (deepfake tools and ChatGPT) were directly controlled and utilized by human actors to execute fraud and generate fake text, with no autonomous decision-making.
• Novelty: Evolved capability. The use of real-time face and voice synthesis to conduct video-call fraud represents a significant technical evolution of traditional social engineering and impersonation scams.

Impact by dimension

• Physical security: Negligible. No physical security threat, kinetic attacks, or critical infrastructure compromise occurred in connection with these incidents.
• Information security: Minor. The use of ChatGPT to spread fake news about a bus crash and deepfakes for fraud represents localized disinformation, but lacks the scale or state-backing of information warfare.
• Sovereignty: Negligible. State authority and core government operations remained unaffected; the incidents prompted regulatory enforcement rather than disrupting constitutional processes.
• Economic security: Minor. While involving a significant financial fraud of 4.3 million yuan against a businessman, the incident was a localized criminal scam rather than a systemic threat to economic security.
• Societal stability: Minor. The spread of fake news about a fatal bus crash and deepfake impersonation caused localized epistemic harm, but did not trigger widespread civil unrest or systemic human rights issues.