Proliferation of Deepfakes Disrupting 2024 Lok Sabha Elections

Digital manipulators in India are using deepfake technology to influence the 2024 Lok Sabha elections. These AI-generated videos and audio clips are designed to tarnish the reputations of political candidates, challenging the integrity of electoral processes.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The widespread, coordinated deployment of millions of AI-generated voice clones and deepfakes to influence the 2024 Indian general election represents a large-scale campaign to manipulate political processes and public opinion.

Additional risk subdomains

• 3.1 False or misleading information: The campaign involved generating fabricated speeches and voice clones that misled voters into believing they were interacting directly with real candidates.
• 1.2 Exposure to toxic content: The report highlights the creation of non-consensual sexualized deepfakes of actress Rashmika Mandanna and negative campaigns designed to degrade opponents.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was driven by human political actors and digital marketing firms intentionally deploying AI deepfake tools to manipulate public opinion and conduct voter outreach.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the widespread deployment of AI-generated deepfakes and voice clones in political campaigns, which falls under the transparency obligations of Limited Risk systems.

AI system and alleged parties

• AI system: DeepFaceLab, SARAL, unspecified
• AI purpose: Deepfake Video Generation; Voice Generation
• Behaviour type: Tool
• Alleged developer: The Indian Deepfaker, The Digital Publicity, Rohit Pal, Obiyan Infotech, Merakii Group
• Alleged deployer: Political candidates of the 2024 Lok Sabha elections
• Alleged harmed parties: Political candidates targeted by deepfakes, Indian electorate, India, Democracy

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Severe. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Severe, indirect Minor
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Severe, indirect Negligible
• Democracy: direct Minor, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Severe, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and malicious content created and spread directly and indirectly by the incident.

Directly caused: Digital manipulators created deepfake videos and manipulated voice calls to portray opposition candidates as foolish or damage their reputations.

Indirectly caused: A viral deepfake of actress Rashmika Mandanna in a provocative outfit was widely circulated, causing a national scandal.

Inferred additional harm: Large volumes of unmonitored, potentially malicious or toxic political deepfakes were likely generated and shared across private messaging networks like WhatsApp.

Civil rights

Reported: The report explicitly describes violations of civil rights related to voter autonomy and the right to free and fair elections.

Directly caused: The deployment of millions of deceptive voice clones without voters' knowledge or consent interferes with their right to make informed political choices.

Inferred additional harm: Widespread manipulation of voters through synthetic media could systematically undermine civil rights related to democratic participation.

Democracy

Reported: The report explicitly describes the erosion of democratic norms caused directly by the incident.

Directly caused: The widespread use of deepfakes, manipulated speeches, and voice clones of deceased politicians to influence the 2024 Indian elections directly erodes democratic norms of transparency and trust.

Inferred additional harm: Continued unlabelled use of AI in elections is likely to cause long-term erosion of public trust in democratic institutions and election outcomes.

Privacy

Reported: The report explicitly describes privacy violations caused directly by the incident.

Directly caused: The reports describe using individuals' photos, videos, and voice recordings to create deepfakes and voice clones without explicit consent, such as using deceased politicians' voices without family permission.

Inferred additional harm: The personal data and biometric likenesses of numerous candidates and citizens were likely harvested and processed without authorization to train these AI models.

Psychological

Reported: The report explicitly describes psychological or reputational distress caused by the non-consensual deepfakes.

Indirectly caused: The viral deepfake of actress Rashmika Mandanna caused public scandal and distress, and political candidates faced targeted reputation damage from manipulated media.

Inferred additional harm: Further psychological distress to other targeted individuals or candidates whose likenesses were manipulated without consent is highly likely.

Epistemic

Reported: The report explicitly describes epistemic harm caused directly by the incident.

Directly caused: AI voice clones and manipulated videos fabricated statements and speeches, making it difficult for voters to distinguish between real and synthetic content.

Inferred additional harm: The massive scale of deepfakes contributes to a broader erosion of shared reality and public consensus, as voters can no longer trust authentic audio or video recordings.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 50000000

Potential causes

Management

• Prioritizing outreach over ethics: Campaign managers prioritize scale and voter impact over ethical concerns.
• Resurrecting deceased politicians: Campaigns use dead leaders' voices without family consent to gain votes.
• Lack of internal oversight: Political parties approve AI campaigns without verifying source authenticity.

Technology

• Realistic Voice Cloning: AI matches voice frequency perfectly, making clones difficult to detect.
• Low-cost AI subscription tools: Cheap apps costing 25 to 30 dollars allow anyone to generate deepfakes.
• Face-swapping software: Software like DeepFaceLab allows easy grafting of faces onto target videos.

Data Inputs

• Available public speech data: Abundant online speeches of politicians are cloned without authorization.
• Lack of original audio reference: No original audio exists to verify or debunk manipulated voice clones.
• High-definition facial data: High-quality photos and videos are easily harvested to build digital avatars.

Human Factors

• Low digital literacy among voters: Voters, especially in rural areas, cannot distinguish AI from real people.
• Overconfidence in AI outreach: Politicians trust AI to build voter rapport without considering deception.
• Malicious intent by campaign managers: Operators deliberately create fake videos and calls to discredit opponents.

Process and Methods

• IP spoofing to avoid detection: Manipulators use external and US IPs to hide their actual location.
• Lack of disclosure in outreach: Campaigns make millions of calls without disclosing that the voice is AI.
• Unsanctioned negative publicity: Firms use deepfakes for political hammering and character assassination.

Regulatory Environment

• Absence of clear AI regulations: Lack of strict rules allows widespread proliferation of political deepfakes.
• Difficult legal enforcement: Anonymity and external IPs make catching digital manipulators very hard.
• Inadequate platform moderation: Social media and messaging apps fail to block or flag deepfake content.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide detailed, first-hand investigative accounts of both unsanctioned negative deepfake campaigns and sanctioned voter outreach efforts, including specific business figures, candidate names, and operational methods.
• Ambiguities identified: The exact scale of the negative deepfake campaigns compared to the sanctioned outreach campaigns is not fully quantified.
• Alternative interpretations: Some might view the sanctioned outreach as a benign technological innovation for language translation rather than a deceptive practice, though the reports highlight the inherent deception.

The widespread deployment of over 50 million AI-generated voice clones and deepfakes during the 2024 Indian elections represents a significant evolution in digital influence operations. While primarily domestic, the scale of this synthetic media campaign poses substantial threats to electoral integrity, information security, and societal trust in democratic institutions.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: India
• Alleged perpetrator: Indian political campaigns and digital marketing firms

Threat characteristics

• Imminence: Long-term. The incident reflects an ongoing, strategic threat to electoral integrity and democratic norms rather than an immediate, acute crisis requiring a 72-hour response.
• Autonomy: Human-controlled. AI systems were used as tools directed by human political campaigns and marketing firms to generate and distribute specific deceptive content.
• Novelty: Evolved capability. Represents a massive scaling and commercialization of existing deepfake technologies, transitioning from isolated incidents to systematic, high-volume election operations.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety threats were reported in connection with these political deepfake campaigns.
• Information security: Substantial. Massive deployment of over 50 million AI-generated voice clones and deepfakes to manipulate voters and discredit political opponents represents a substantial information operations threat.
• Sovereignty: Substantial. Widespread use of deceptive synthetic media directly targeted the 2024 Lok Sabha electoral process, threatening the integrity of democratic decision-making at a national scale.
• Economic security: Negligible. While representing a growing commercial market for synthetic media, the incident did not directly threaten India's economic stability or strategic technological infrastructure.
• Societal stability: Substantial. Large-scale social manipulation targeting tens of millions of citizens with unconsented biometric clones risks eroding public trust and undermining the right to free and fair elections.