Purported Deepfake of Andrew Forrest Used to Promote Fraudulent 'Quantum AI' Crypto Platform on Facebook

Cybertrace reportedly identified a purported deepfake of Australian billionaire Andrew Forrest circulating on Facebook on January 27, 2024, falsely depicting him endorsing a fraudulent crypto platform called Quantum AI. The video reportedly altered footage from a 2023 Rhodes Trust talk. Forrest reportedly condemned Meta for failing to curb AI-generated frauds.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involves the use of AI-generated deepfakes of a public figure to promote a fraudulent cryptocurrency platform, directly targeting individuals for financial scamming.

Additional risk subdomains

• 3.1 False or misleading information: The deepfake video generated and spread false information regarding Andrew Forrest's endorsement of a cryptocurrency platform.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The deepfake scam was intentionally created and deployed by human malicious actors using post-deployment AI generation tools to defraud social media users.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the use of 'Artificial intelligence (AI) generated deepfake videos' which fall under the category of AI-generated content requiring transparency obligations under the Act.

AI system and alleged parties

• AI system: unspecified
• AI purpose: Deepfake Video Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: Unknown voice cloning technology developers, Unknown deepfake technology developers
• Alleged deployer: Unknown scammers impersonating Andrew Forrest, Unknown scammers, Quantum AI
• Alleged harmed parties: Truth, Investors, General public of Australia, General public, Facebook users, Epistemic integrity, Australian investors, Andrew Forrest

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the report describes the creation and spread of a deepfake video designed to defraud users.

Directly caused: An AI-modified deepfake video of Andrew Forrest was created and uploaded to Facebook to promote a fraudulent crypto platform.

Inferred additional harm: It is likely that similar malicious deepfake ads continue to be generated and spread across Meta's platforms, exposing thousands of additional users to fraudulent content.

Epistemic

Reported: Yes, the report notes that deepfakes hijacking images of public figures erode trust in the media.

Directly caused: The deepfake video fabricated an endorsement by Andrew Forrest, misleading viewers into believing he supported the fraudulent platform.

Indirectly caused: The proliferation of such deepfakes contributes to a broader erosion of trust in social media and digital news content.

Inferred additional harm: The ongoing spread of highly realistic deepfakes is likely to further degrade public trust in video evidence and media authenticity.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 10000

Potential causes

Management

• Inadequate Safety Enforcement: Meta fails to enforce its 2020 deepfake ban effectively on its platforms.

Technology

• Advanced Deepfake Generation: AI tools easily alter actions and mannerisms to create realistic fakes.
• Lagging Deepfake Detection: Verification systems fail to keep pace with evolving AI generation tools.

Data Inputs

• Use of Real Public Footage: Scammers source legitimate videos of public figures to train AI models.

Human Factors

• Trust in Public Figures: Unassuming users trust recognizable figures like prominent businessmen.

Process and Methods

• Ineffective Ad Screening: Social media platforms fail to filter out deceptive and doctored video ads.
• Slow Content Removal: Platforms take days to remove scams, allowing them to reach many targets.

Regulatory Environment

• Lack of Platform Liability: Social media companies face minimal legal liability for negligent ad platforms.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly describes the nature of the deepfake, the platform it was hosted on, the target individual, and the fraudulent intent. There is little ambiguity regarding the classification of this event as a deepfake-enabled financial scam.
• Ambiguities identified: The specific AI tool used to generate the deepfake is not identified, and the exact number of victims or financial losses from this specific ad is not quantified.

A deepfake-enabled financial scam targeting Australian billionaire Andrew Forrest on Facebook. While representing an ongoing challenge for platform moderation and contributing to epistemic erosion, the incident is a localized criminal fraud with minor national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding synthetic media fraud rather than an active national security crisis requiring immediate intervention.
• Autonomy: Human-controlled. The AI was used as a tool by human actors who directed the creation and dissemination of the fraudulent content.
• Novelty: Established threat. Deepfake-enabled financial scams targeting prominent figures on social media are an established and frequently observed threat.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were affected or threatened.
• Information security: Minor. Involves synthetic media targeting a prominent figure, but lacks state-actor backing or geopolitical targeting, limiting its impact to standard cybercrime.
• Sovereignty: Negligible. No government functions, electoral systems, or sovereign state authorities were targeted or affected.
• Economic security: Minor. Presents minor economic impact through localized fraudulent activity targeting retail investors, but poses no systemic threat to national financial systems.
• Societal stability: Minor. Contributes to the gradual erosion of public trust in digital media and social platforms, but does not trigger large-scale civil unrest or systematic human rights abuses.