Instagram Video Featured Deepfake Audio of US President Making Transphobic Remarks

A deepfaked audio of US President Joe Biden making transphobic remarks played on top of a video showing him giving a speech was released on Instagram and circulated on social media.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The incident involved the intentional creation and viral dissemination of a political deepfake of President Biden to spread disinformation and influence public opinion.

Additional risk subdomains

• 1.2 Exposure to toxic content: The deepfake generated and exposed social media users to highly toxic, transphobic hate speech and references to suicide.
• 4.3 Fraud, scams, and targeted manipulation: The AI tool was used to impersonate a public figure by cloning his voice without consent to manipulate viewers.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by a human actor intentionally using a deployed AI voice cloning tool to generate and distribute a deceptive, transphobic deepfake video of President Biden.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is used to generate AI-generated content such as deepfakes, which under the EU AI Act falls under Limited Risk, requiring specific transparency obligations to ensure users are informed that the content is AI-generated.

AI system and alleged parties

• AI system: ElevenLabs voice synthesis platform (ElevenLabs)
• AI purpose: Voice Generation; Deepfake Video Generation
• Behaviour type: Tool
• Alleged developer: unknown
• Alleged deployer: unknown
• Alleged harmed parties: transgender people, President Joe Biden

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes a deepfake video containing transphobic hate speech and references to suicide.

Directly caused: An AI-generated video of Joe Biden making transphobic statements ('You will never be a real woman', 'homosexual man twisted by drugs') was created and shared on Instagram and Facebook.

Indirectly caused: The video was shared widely, gaining over 12,000 likes on one Instagram post and amassing hundreds of thousands of views across social media platforms.

Inferred additional harm: Further spread of the toxic audio and video across other unmonitored online forums and messaging groups, exposing more users to hate speech.

Democracy

Reported: The reports discuss the threat of deepfakes to political processes and the ease of generating disinformation-filled videos of political leaders.

Directly caused: A deepfake of President Biden was created to misrepresent his political stance on transgender rights.

Indirectly caused: The video misled some social media users, contributing to the erosion of trust in public figures and political communication.

Inferred additional harm: Increased public skepticism and erosion of democratic discourse as citizens find it harder to distinguish real political speeches from AI-generated fakes.

Psychological

Reported: The report explicitly describes that the video contained harsh criticism of transgender women, including a reference to suicide, which shocked and distressed users.

Indirectly caused: The transphobic remarks and suicide references likely caused emotional distress and anxiety among transgender individuals who viewed the viral video.

Inferred additional harm: Widespread distress and psychological harm to members of the transgender community exposed to the viral hate speech, though unquantified.

Epistemic

Reported: The reports explicitly describe the video as a 'doctored version' and 'disinformation-filled' that misled some users.

Directly caused: The AI synthesized highly realistic fake audio of Joe Biden making statements he never made, paired with manipulated lip movements.

Indirectly caused: The video amassed hundreds of thousands of views, with some users believing it was real or not knowing what to believe, degrading the shared information ecosystem.

Inferred additional harm: Widespread confusion and a general decline in the perceived reliability of video and audio evidence online.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 100000

Potential causes

Management

• Premature Technology Release: Management released the tool to the public before implementing safety controls.
• Lack of Prior Risk Assessment: The startup deployed the model without anticipating voice cloning misuse.

Technology

• Advanced Voice Synthesis: ElevenLabs beta allowed realistic voice cloning with minimal audio samples.
• Open-source Lip-sync Algorithms: Free tools enabled matching mouth movements to synthetic audio.

Data Inputs

• Public Voice Samples: Abundant public audio of Biden was used to train the voice-cloning model.
• Source Video Footage: Real footage from a Jan 25 speech was used to map mouth movements.

Human Factors

• Malicious Intent by Creators: Bad actors intentionally generated and spread transphobic disinformation.
• User Susceptibility: Some social media users believed the highly realistic deepfake was authentic.

Process and Methods

• Unrestricted Beta Release: ElevenLabs initially allowed anonymous, free access without safeguards.
• Reactive Content Moderation: Social media platforms relied on post-facto flagging and fact-checking.

Regulatory Environment

• Lack of Deepfake Regulation: No legal frameworks existed to restrict the generation of deceptive deepfakes.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the deepfake video, the specific AI tool used (ElevenLabs), the platform safeguards implemented in response, and expert analysis of the digital manipulation techniques. There is no conflicting information regarding the fake nature of the video or the technology used.
• Ambiguities identified: The exact identity of the creator is unknown, and the precise number of total views across all platforms is described generally as 'hundreds of thousands' rather than a specific figure.
• Alternative interpretations: None. The video is universally confirmed by multiple fact-checkers and forensics experts to be an AI-generated deepfake.

An anonymous actor used ElevenLabs' voice cloning tool to create a transphobic deepfake of U.S. President Joe Biden. While the video circulated widely on social media, it was quickly debunked by fact-checkers, resulting in minor information security and societal impacts without threatening critical infrastructure or government functions.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The immediate incident has been resolved and debunked, representing an ongoing strategic concern regarding synthetic media rather than an active crisis.
• Autonomy: Human-controlled. The AI voice cloning and lip-synching tools were directly operated and controlled by human actors to generate the content.
• Novelty: Evolved capability. Represents an evolution in the ease and realism of voice-cloning technology available to the public, building on existing deepfake methods.

Impact by dimension

• Physical security: Negligible. No physical infrastructure, systems, or human safety threats were involved or impacted.
• Information security: Minor. The deepfake targeted the U.S. President to spread disinformation on social media, but it was quickly debunked by fact-checkers and lacked evidence of a coordinated state-sponsored campaign.
• Sovereignty: Negligible. While the video depicted the head of state, it did not disrupt government operations, decision-making, or constitutional processes.
• Economic security: Negligible. No financial systems, strategic technologies, or economic supply chains were targeted or affected.
• Societal stability: Minor. The video spread transphobic hate speech and caused distress within the transgender community, but did not lead to large-scale civil unrest.