Alleged Deepfake Audio of Imran Khan Calls for Election Boycott, Misleading Pakistan Voters

A purported deepfake audio clip, falsely attributed to Imran Khan urging a PTI (Pakistan Tehreek-e-Insaf) election boycott, circulated on social media on the eve of Pakistan's general elections. This sophisticated AI-generated misinformation aimed to mislead voters, highlighting the growing challenge of digital manipulation in political discourse.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The deepfakes were used to conduct a coordinated disinformation campaign aimed at manipulating public opinion and the political process during Pakistan's national elections.

Additional risk subdomains

• 3.1 False or misleading information: The AI-generated deepfakes spread false information regarding political candidates boycotting the election.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was driven by human actors who intentionally generated and disseminated deepfake audio and video clips post-deployment to mislead voters.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the use of AI-generated content such as deepfakes, which falls under Risk Level 3 due to transparency and disclosure requirements.

AI system and alleged parties

• AI system: unspecified
• AI purpose: Voice Generation; Deepfake Video Generation
• Behaviour type: Tool
• Alleged developer: unknown
• Alleged deployer: Unknown social media accounts
• Alleged harmed parties: Voters in Pakistan, PTI (Pakistan Tehreek-e-Insaf), Imran Khan, Democracy

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes malicious deepfake content created and spread directly by the incident.

Directly caused: AI-generated deepfake audio of Imran Khan and deepfake videos of other PTI candidates were created and spread on social media platforms like X.

Inferred additional harm: It is likely that other unmentioned deepfake videos and audios of political figures were generated and circulated during the election period.

Civil rights

Reported: The report explicitly describes potential impacts on the civil rights of voters to access truthful information during an election.

Indirectly caused: The dissemination of deepfakes threatened the civil rights of Pakistani voters by attempting to deceive them into believing their preferred candidates had boycotted the election.

Democracy

Reported: The report explicitly describes the erosion of democratic norms and threats to the integrity of the electoral process.

Indirectly caused: The deepfakes threatened the integrity of the electoral process and the quality of democracy by potentially duping voters into making decisions based on false information.

Inferred additional harm: The widespread use of deepfakes in elections likely causes long-term public distrust in democratic institutions and election outcomes.

Epistemic

Reported: The report explicitly describes epistemic harm through the spread of AI-generated misinformation and fabricated media.

Directly caused: AI-generated fake audio and videos fabricated statements of political leaders announcing election boycotts, misleading the public.

Indirectly caused: The fabrication of these statements forced political parties and candidates to spend critical time and resources debunking the false claims.

Inferred additional harm: The incident likely contributed to a broader erosion of shared truth and increased skepticism among the electorate regarding the authenticity of any digital media.

People affected

• Occurrences reported: 4
• People reportedly harmed: 3
• People reportedly exposed: 128000000

Potential causes

Management

• Growth Prioritization: Social media platforms prioritized engagement over information integrity.
• Poor Incident Response: Management lacked proactive strategies to counter election-disrupting deepfakes.

Technology

• Sophisticated AI Voice Cloning: Generates highly realistic but artificial audio of political leaders.
• Post-Production Manipulation: Unnatural noise was added to simulate authenticity and mask AI generation.
• Lack of Real-Time Detection: Platforms lacked automated tools to instantly detect and flag AI deepfakes.

Data Inputs

• Unlabeled Synthetic Audio: Deepfakes lacked metadata or watermarks indicating they were AI-generated.
• Lack of Source Verification: No verification of the audio source before dissemination on social media.

Human Factors

• Voter Susceptibility: Public likely to make voting decisions based on unverified false audio.
• Overreliance on Social Media: Users trust and share viral content without verifying its authenticity.

Process and Methods

• Delayed Fact-Checking: Manual verification processes could not keep up with viral dissemination.
• Internet Shutdowns: Government internet cuts blocked journalists from debunking fake news quickly.
• Ineffective Platform Moderation: Social media platforms failed to promptly remove flagged deceptive content.

Regulatory Environment

• Weak Legal Frameworks: Pakistan lacked specific criminalization for malicious AI-generated content.
• Inadequate Regulatory Oversight: Election and media authorities failed to enforce strict codes of conduct.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent, and detailed accounts of the deepfake incidents, including expert verification of the synthetic audio and statements from the affected political party.
• Ambiguities identified: The specific AI models or platforms used to generate the deepfakes are not identified.

Malicious actors deployed AI-generated deepfake audio and video of political leaders in Pakistan, falsely claiming an election boycott on the eve of the 2024 general elections. The incident represents a substantial threat to sovereignty and information security by targeting democratic processes, though it was quickly mitigated by public debunking.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: Pakistan
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Immediate. The deepfakes were released on the eve of the election, requiring immediate debunking within a 24-to-48-hour critical window.
• Autonomy: Human-controlled. The AI systems were used as tools by human operators who generated and manually disseminated the deepfake media.
• Novelty: Evolved capability. While deepfakes have been used before, the rapid generation and deployment of high-fidelity voice clones in a national election represents an evolved threat.

Impact by dimension

• Physical security: Negligible. No physical security threats or critical infrastructure disruptions were reported in connection with this incident.
• Information security: Substantial. AI-generated deepfakes were deployed in a coordinated disinformation campaign targeting political leaders to mislead voters on the eve of a national election.
• Sovereignty: Substantial. The incident directly targeted the integrity of Pakistan's democratic electoral process by attempting to manipulate voter behavior through fabricated candidate boycotts.
• Economic security: Negligible. No significant economic or technological security impacts were reported as a result of this incident.
• Societal stability: Substantial. The widespread dissemination of political deepfakes threatened societal stability by eroding public trust in the authenticity of election-related information.