AI Deepfakes for Voter Outreach Flood Indian Elections

During the 2024 Indian elections, politicians used AI-generated deepfakes to reach voters, who might be unaware they're interacting with digital clones. Providers like Divyendra Singh Jadoun of Polymath Synthetic Media Solutions created deepfakes for personalized messages. This practice, used by various political parties, is not truthful, as voters may be misled by AI-generated content posing as genuine interactions with political figures.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The incident involves the coordinated, large-scale deployment of AI-generated voice clones and video avatars to influence voters and manipulate political processes during an election.

Additional risk subdomains

• 3.1 False or misleading information: The AI voice clones generate synthetic speech that misleads voters into believing they are having real, personalized conversations with political candidates.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The deployment of AI voice clones and video avatars was an intentional decision by human political campaigns and consultants to influence voters during the post-deployment phase of the election.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The incident involves chatbots and AI-generated content such as deepfakes, which under the EU AI Act are subject to transparency obligations to ensure users are aware they are interacting with AI.

AI system and alleged parties

• AI system: DeepFaceLab, Mistral, unspecified AI voice cloning (Mistral AI)
• AI purpose: Voice Generation; Deepfake Video Generation
• Behaviour type: Multi-agent
• Alleged developer: Sumit Savara, Sagar Vishnoi, Polymath Synthetic Media Solutions, iToConnect, IndiaSpeaks Research Lab, Divyendra Singh Jadoun
• Alleged deployer: Y. S. Jagan Mohan Reddy, Ram Chandra Choudhary, Prem Singh Tamang, Indian National Congress (INC), Bharatiya Janata Party (BJP)
• Alleged harmed parties: Truth, Political integrity and election fairness, Indian voters, General public misled by deepfake content, Democracy

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Minor, indirect Minor
• Privacy: direct Severe, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Severe, indirect Severe
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly mentions the potential for toxic or malicious content, noting that political parties have used AI to warp reality through cheap audio fakes, propaganda images, and parodies.

Directly caused: The report notes that Jadoun created AI-generated propaganda songs for several politicians, and that other unsanctioned deepfakes targeted political opponents.

Inferred additional harm: It is likely that malicious actors used similar voice-cloning technologies to spread highly toxic or defamatory content targeting opponents during the election, though specific details are limited.

Democracy

Reported: The report explicitly describes concerns regarding the erosion of democratic norms due to the scale of undisclosed AI-generated political outreach.

Directly caused: Political campaigns deployed over 50 million AI-generated voice clone calls to voters without disclosing that they were interacting with a machine, potentially manipulating voter opinion.

Indirectly caused: The use of resurrected dead politicians to endorse candidates and the sharing of illegal voice clone endorsements by celebrities led to warnings from India's election watchdog.

Inferred additional harm: The normalization of undisclosed synthetic interactions in elections is likely to permanently degrade public trust in democratic processes and candidate communications.

Privacy

Reported: The report explicitly mentions privacy-related concerns regarding the use of voter data to target personalized calls.

Directly caused: Politicians used data from the SARAL app, which tracks beneficiaries of government schemes, to target personalized AI calls directly to voters' phones.

Inferred additional harm: The integration of personal voter databases with synthetic voice generation systems likely violated the privacy expectations of millions of citizens whose data was used without explicit consent.

Epistemic

Reported: The report explicitly describes epistemic harm, highlighting how deepfakes and voice clones warp reality and deceive voters.

Directly caused: Voters received personalized calls believing they were speaking directly with real candidates, unaware that they were interacting with AI-generated voice clones.

Indirectly caused: The proliferation of audio deepfakes makes it difficult to verify authentic statements, as there is often no original reference audio to compare against.

Inferred additional harm: The widespread use of synthetic media in politics is likely to lead to a broader 'liar's dividend' where politicians can falsely claim real, damaging recordings of themselves are deepfakes.

People affected

• Occurrences reported: 1
• People reportedly exposed: 50000000

Potential causes

Management

• Prioritizing Scale Over Safety: Firms and politicians prioritize massive outreach over AI safety and consent.

Technology

• AI Agent Hallucinations: LLM generated incorrect replies, loops, and screaming audio responses.
• High Latency in AI Agent Calls: Delays in machine responses disrupted natural back-and-forth conversation.
• Flawed Voice Synthesis: Cloned voices had anglicized pronunciations and unnatural tones.

Data Inputs

• Lack of Voice Data Consent: Using speeches of deceased politicians without family consent to clone voices.
• Misuse of Target Voter Data: Political apps collect and exploit personal data for hyper-targeted AI calls.

Human Factors

• Voter Deception Vulnerability: Less tech-savvy rural voters fail to realize they are speaking to AI clones.
• Lack of AI Literacy: Politician candidates and voters do not understand how AI cloning works.

Process and Methods

• Absence of AI Disclosure: Campaigns deploy deepfakes without labeling them as AI-generated content.
• Inadequate Testing of AI Agents: Deploying live interactive AI agents to voters without proper quality checks.

Regulatory Environment

• Lack of Deepfake Regulations: Weak regulatory oversight allows unchecked deployment of political voice clones.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides extensive, detailed, and first-hand journalistic accounts of the deployment of AI voice clones and deepfakes in the 2024 Indian elections, including interviews with developers, consultants, and politicians. The scale of the deployment (over 50 million calls) and the specific technologies used (DeepFaceLab, Mistral) are clearly documented.
• Ambiguities identified: The exact proportion of calls that were fully interactive AI agents versus simple one-way prerecorded voice clones is not fully quantified, though the report suggests interactive calls were mostly experimental due to technical limitations.
• Alternative interpretations: Some might interpret these AI campaigns as benign voter engagement and translation tools rather than malicious political manipulation, though the lack of disclosure strongly points to epistemic and democratic risks.

During the 2024 Indian elections, political parties deployed over 50 million undisclosed AI voice clones and video avatars for personalized voter outreach. While not a kinetic threat, the unprecedented scale of synthetic media targeting democratic processes represents a substantial information warfare and societal stability concern that could undermine public trust in democratic institutions.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: India
• Alleged perpetrator: Indian political parties and consultants

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding the normalization and scale of synthetic media in democratic processes.
• Autonomy: Human-supervised. AI systems generated personalized content and handled interactive calls autonomously, but under the strategic direction and setup of human campaign administrators.
• Novelty: Evolved capability. Represents a significant scale and sophistication advancement of existing voice-cloning and deepfake technologies applied to political campaigns.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic targeting systems were impacted by this incident.
• Information security: Substantial. Massive deployment of over 50 million undisclosed AI-generated voice clones and deepfakes targeting voters represents a substantial evolution in automated information manipulation during a national election.
• Sovereignty: Substantial. The scale of undisclosed synthetic media targeting voters potentially undermines democratic processes and electoral integrity, warranting attention from security and electoral agencies.
• Economic security: Negligible. No significant threats to financial systems, critical supply chains, or strategic technological competitive advantages were reported.
• Societal stability: Substantial. Widespread deployment of synthetic voice clones without consent for population-scale political targeting risks eroding public trust and manipulating social cohesion.