53% of American and British Businesses Report Attacks by AI-Powered Deepfake Scams in 2024

According to Medius, Deepfake scams have targeted 53% of businesses in the U.S. and U.K., with 43% falling victim. Using AI to create realistic fake videos and audio of corporate executives, scammers have successfully stolen millions, including $25 million from British engineering group Arup.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involved scammers using AI-generated deepfakes to impersonate company executives and fraudulently siphoning $25 million from Arup.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The employee relied on the visual and auditory representation of the CFO during the video call, demonstrating trust in the technology that led to unsafe financial transactions.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was initiated by human malicious actors who intentionally deployed generative AI deepfakes to execute a financial scam after the models were trained and deployed.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the use of 'AI-generated content such as deepfakes', which falls under the Limited Risk category requiring specific transparency obligations to ensure users are informed.

AI system and alleged parties

• AI system: unspecified
• AI purpose: Deepfake Video Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: Unknown deepfake technology developers
• Alleged deployer: Unknown scammers, Unknown deepfake creators
• Alleged harmed parties: Finance professionals, Employees, British businesses, Arup, American businesses

Harm severity

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Substantial, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report describes 1 incident causing total financial loss of $25000000.

Directly caused: Average financial loss per occurrence: $25000000. Scammers successfully siphoned $25 million from the engineering firm Arup.

Indirectly caused: N/A

Inferred additional harm: While WPP was targeted unsuccessfully, the broader survey indicates 43% of businesses fell victim to deepfake scams, implying widespread unquantified financial losses across other organizations.

Epistemic

Reported: Yes, the report describes the use of highly convincing AI-generated fake video and audio to deceive employees.

Directly caused: Scammers used deepfake technology to pose as the organization's CFO and other employees in a video conference, creating a false reality.

Indirectly caused: N/A

Inferred additional harm: The proliferation of such highly convincing deepfakes could lead to a broader erosion of trust in digital communications and video conferencing within corporate environments.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Inadequate Risk Assessment: Failure to assess and prepare for generative AI threats to financial security.
• Insufficient Security Training Budget: Lack of investment in educating high-risk departments on deepfakes.

Technology

• Sophisticated Deepfake Generation: AI tools can convincingly clone executive voices and faces.
• Inadequate Verification Tools: Traditional security systems cannot detect AI-driven vocal anomalies.
• Dark Web Software Democratization: Cheap, accessible scamming software is widely available on the dark web.

Data Inputs

• Public Executive Media Availability: YouTube videos and podcasts provide source material for cloning.
• Lack of Biometric Data Controls: Absence of secure digital identity wallets to verify participant liveness.

Human Factors

• Inherent Trust in Familiar Voices: Employees naturally trust voices and faces they recognize.
• Susceptibility to Social Engineering: Pressure from false urgency and flattery compromises decision-making.
• Lack of Deepfake Awareness: Staff lack training to recognize and respond to AI-driven impersonations.

Process and Methods

• Weak Payment Verification Processes: Absence of multi-level sign-offs or dual-authorization for transfers.
• Inadequate Incident Response Plans: Lack of documented procedures for responding to suspected deepfake attacks.
• Reliance on Verbal Communication: Critical financial transactions conducted over unverified voice/video calls.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, detailed, and consistent accounts of the $25 million deepfake scam at Arup, including the specific mechanism of the fraud (video conference with deepfake CFO and employees) and the broader context of deepfake financial scams.
• Ambiguities identified: The specific generative AI models used by the scammers are not identified.

A sophisticated generative AI deepfake scam defrauded engineering firm Arup of $25 million by impersonating company executives on a video call. This incident highlights the growing systemic threat that advanced synthetic media poses to corporate financial security and economic stability in the US and UK.

• Overall national security impact: Substantial
• Response level: Substantial
• Scope: Multiple nations
• Primary target: United States and United Kingdom
• Other affected: Hong Kong
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing, systemic strategic concern regarding corporate security rather than an active, time-critical national security crisis.
• Autonomy: Human-controlled. The AI was used as a tool by human scammers who directed the creation and deployment of the deepfake assets.
• Novelty: Evolved capability. Represents a highly sophisticated evolution of existing business email compromise and phishing tactics using generative AI.

Impact by dimension

• Physical security: Negligible. No physical infrastructure, kinetic assets, or human safety systems were compromised or targeted in this financial fraud incident.
• Information security: Minor. Advanced deepfake technology was used to deceive employees, representing a significant advancement in synthetic media capabilities, but without targeting national intelligence assets.
• Sovereignty: Negligible. The incident targeted private corporate entities and did not impact state authority, electoral processes, or core government functions.
• Economic security: Substantial. Widespread deepfake financial fraud targeting major corporations like Arup ($25 million loss) poses a systemic threat to corporate financial security and economic stability.
• Societal stability: Negligible. The incident was a targeted financial fraud and did not involve mass surveillance, civil liberties violations, or threats to social cohesion.