Deepfake Voice Exploit Compromises Retool's Cloud Services

In August 2023, a hacker reportedly was successful in breaching Retool, an IT company specializing in business software solutions, impacting 27 cloud customers. The attacker appears to have initiated the breach by sending phishing SMS messages to employees and later used an AI-generated deepfake voice in a phone call to obtain multi-factor authentication codes. The breach seems to have exposed vulnerabilities in Google's Authenticator app, specifically its cloud-syncing function, further enabling unauthorized access to internal systems.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involved a hacker using an AI-powered voice deepfake to impersonate an IT staff member, successfully executing a targeted social engineering scam to gain unauthorized access.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The breach resulted in unauthorized access to GSuite accounts and internal systems, compromising 27 cloud customers.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The security breach was initiated and executed by a human hacker who intentionally used social engineering and an AI voice deepfake to compromise Retool's systems.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The incident involves the use of AI-generated content, specifically an AI-powered voice deepfake, which falls under the category of deepfakes requiring transparency obligations.

AI system and alleged parties

• AI system: AI-powered deepfake
• AI purpose: Voice Generation
• Behaviour type: Tool
• Alleged developer: unknown
• Alleged deployer: Unknown hacker
• Alleged harmed parties: Retool employee who was the victim of the unknown hacker, Retool, Google, 27 of Retool's clients

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Minor
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report explicitly describes a security breach compromising GSuite accounts and internal systems.

Directly caused: The attacker gained unauthorized access to GSuite accounts and MFA tokens of a Retool employee.

Indirectly caused: The breach ensnared and compromised the accounts of 27 cloud customers.

Inferred additional harm: It is highly likely that sensitive personal or corporate data belonging to the 27 cloud customers was accessed or exposed during the period of unauthorized access.

People affected

• Occurrences reported: 1
• People reportedly harmed: 28
• People reportedly exposed: 37

Potential causes

Management

• Inadequate Phishing Training: Training failed to prepare staff for AI-driven vishing attacks.
• Unmonitored MFA Configuration: Management failed to restrict or audit Google Authenticator cloud sync.

Technology

• AI Voice Deepfake Generation: Attacker generated highly realistic voice clone of IT team member.
• Authenticator Cloud Syncing: Google Authenticator synced MFA codes to compromised Google account.

Data Inputs

• Reconnaissance Data on Staff: Attacker used specific details about coworkers and office layouts.
• Employee Voice Samples: Attacker sourced voice recordings to train the AI deepfake model.

Human Factors

• Vulnerability to Social Engineering: Employee trusted the realistic deepfaked voice and shared MFA code.
• Phishing Link Interaction: Employee clicked a malicious link in an SMS pretending to be IT.

Process and Methods

• Inadequate Voice Verification: No process to verify identity of callers claiming to be IT staff.
• MFA Code Sharing via Phone: Lack of strict protocols preventing sharing of MFA codes over calls.

Regulatory Environment

• Lack of Deepfake Regulations: Absence of strict controls on commercial voice cloning technologies.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides a clear, detailed account of the attack vector, the role of the voice deepfake, and the specific vulnerability in Google Authenticator's cloud sync that enabled the breach.
• Ambiguities identified: The specific AI voice cloning tool used by the hacker is not identified.
• Alternative interpretations: None. The facts of the social engineering attack and subsequent breach are straightforward.

A security breach at Retool where an attacker used an AI-generated voice deepfake to impersonate an IT employee, bypassing multi-factor authentication and compromising 27 cloud customers. The incident highlights the growing threat of AI-enabled social engineering to corporate cybersecurity, though its direct national security impact remains minor.

• Overall national security impact: Minor
• Scope: Unknown
• Primary target: Unknown
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The active breach has been resolved. The incident represents an ongoing strategic concern regarding the evolution of AI-enabled social engineering tactics.
• Autonomy: Human-controlled. The AI voice cloning system was directly controlled and deployed by a human attacker as a tool within a manual social engineering campaign.
• Novelty: Evolved capability. While social engineering is an established threat, the integration of highly realistic AI voice deepfakes to impersonate specific internal staff represents an evolved and highly sophisticated capability.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety threats were reported in this incident.
• Information security: Minor. The incident involved a targeted social engineering attack using an AI voice deepfake to bypass corporate MFA. While it compromised internal systems, it did not target intelligence systems or state-level information security.
• Sovereignty: Negligible. The breach affected a private IT company and its corporate customers, with no reported impact on government functions, elections, or state sovereignty.
• Economic security: Minor. The compromise of Retool and 27 cloud customers represents a localized economic and technological risk to private entities, but does not threaten strategic national industries or critical financial systems.
• Societal stability: Negligible. The incident compromised corporate and employee privacy on a limited scale, but did not threaten societal stability, civil liberties, or broader human rights.