Scams Reportedly Impersonating Wealthy Investors Proliferating on Facebook

Scams are reportedly proliferating throughout Facebook impersonating wealthy individuals such as Bill Ackman, Cathie Wood, Steve Cohen, Peter Lynch, and Ray Dalio. In some cases, it seems deepfake technology is being employed, while simultaneously Facebook's own AI systems are allegedly faltering in their ability to halt the spread of these fraudulent ads despite being reported.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involves malicious actors using deepfakes and targeted ads to execute financial scams and defraud specific victims for personal gain.

Additional risk subdomains

• 7.3 Lack of capability or robustness: Meta's automated AI moderation systems failed to effectively detect, block, or remove the fraudulent ads despite repeated user reports.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The primary cause of the risk is the intentional decision by human scammers to deploy deepfakes and fraudulent ads to deceive investors.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: The incident involves the use of AI-generated content, specifically deepfakes, which falls under Limited Risk and requires transparency obligations.

AI system and alleged parties

• AI system: Facebook AI detection tools (Meta)
• AI purpose: Deepfake Video Generation; Ad Delivery
• Behaviour type: Tool
• Alleged developer: Meta, Facebook
• Alleged deployer: Unknown scammers, Meta, Facebook
• Alleged harmed parties: Steve Cohen, Ray Dalio, Peter Lynch, Peter Bourget, Investors, General public, Cathie Wood, Bill Ackman

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Substantial
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the report describes total losses of 3.2 million dollars for a Reddit group, 1.3 million dollars for a couple, and 45,000 dollars for an individual.

Indirectly caused: Victims lost millions of dollars, with an average loss of 4.5 million dollars across the reported groups and individuals in this continuous incident.

Inferred additional harm: Total financial losses across all victims of these specific deepfake scams likely exceed tens of millions of dollars.

Malicious content

Reported: Yes, the report describes the creation and spread of deepfake videos and fraudulent ads.

Directly caused: Scammers generated deepfake videos of financial experts and distributed them as ads on Facebook.

Inferred additional harm: Thousands of other deepfake ads and scam messages were likely generated and distributed across Meta's platforms.

Psychological

Reported: Yes, the report describes victims experiencing embarrassment and severe stress due to losing their savings.

Indirectly caused: Victims suffered significant emotional distress, embarrassment, and anxiety after being defrauded of their life savings.

Inferred additional harm: Many of the hundreds of unnamed victims likely suffered similar psychological distress and anxiety due to sudden financial ruin.

Epistemic

Reported: Yes, the report describes the use of deepfakes to fabricate statements and mislead users.

Directly caused: Deepfake videos fabricated statements of financial experts to deceive users into believing they were endorsing scams.

Inferred additional harm: The proliferation of realistic deepfakes erodes public trust in online financial information and video evidence.

People affected

• Occurrences reported: 1
• People reportedly harmed: 32
• People reportedly exposed: 10000

Potential causes

Management

• Staff Reductions in Safety Teams: Job cuts at Meta reduced human oversight for AI moderation systems.
• Prioritizing Ad Revenue: Platform allowed unverified high-risk financial ads to run for weeks.

Technology

• Deepfake Video Generation: Sophisticated AI generated deepfakes of financial figures deceived victims.
• Targeting Algorithm Exploitation: Look-alike audience algorithms optimized delivery to vulnerable users.
• Inadequate AI Detection Tools: Meta's AI failed to proactively flag and block celebrity-bait scam ads.

Data Inputs

• Missing Identity Verification Data: Lack of verified identity inputs allowed scammers to buy ads as celebrities.

Human Factors

• User Trust in AI Deepfakes: Victims believed realistic AI-generated videos of trusted financial figures.
• User Vulnerability to Scams: Older and wealthier users lacked digital literacy to spot AI-generated fakes.

Process and Methods

• Flawed Automated Moderation: Meta's automated review falsely cleared reported scam ads as compliant.
• Inadequate Reporting Loop: User reports of scam ads failed to trigger rapid manual or AI reviews.

Regulatory Environment

• Section 230 Liability Shield: Web platforms have legal immunity for deceptive content created by others.

Information quality

• Classification confidence: High
• Reason for confidence: The Wall Street Journal report provides detailed, credible accounts of specific victims, financial figures targeted, and the mechanisms of the scams, including the use of deepfakes and Facebook's ad platform.
• Ambiguities identified: The exact AI tools used to generate the deepfakes are not specified.
• Alternative interpretations: None.

Sophisticated financial scams on Facebook utilized deepfake videos of prominent financial figures to defraud retail investors of millions of dollars. While the incident highlights the growing threat of synthetic media and the limitations of automated moderation systems, its national security impact remains minor as it does not target critical infrastructure, state sovereignty, or national economic stability.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The proliferation of deepfake scams represents an ongoing strategic concern and regulatory challenge rather than an immediate national security crisis.
• Autonomy: Human-controlled. AI was utilized as a tool for generating synthetic media and targeting ads, but human operators directed the fraudulent campaigns and executed the scams.
• Novelty: Evolved capability. The incident demonstrates a significant advancement in the realism and deployment scale of deepfake technology for financial fraud, moving beyond basic phishing.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure damage were reported in connection with these financial scams.
• Information security: Minor. Deepfake technology was used to impersonate private financial figures for fraud, showing an evolved capability in synthetic media, though not targeting state intelligence or institutions.
• Sovereignty: Negligible. No government decision-making, electoral systems, or core sovereign functions were disrupted or compromised by these activities.
• Economic security: Minor. The incident caused substantial individual financial losses totaling millions of dollars, but did not threaten national macroeconomic stability, strategic industries, or tech leadership.
• Societal stability: Minor. The scams caused psychological distress and financial harm to targeted individuals, but did not threaten large-scale social cohesion, civil liberties, or result in civil unrest.