Deepfakes Circulating and Eroding Electoral Integrity in the Lead-up to 2024 South Korean legislative election

In the lead-up to Korea's parliamentary elections, at least 129 deepfake videos and images were reported to have been detected, violating new election laws. These AI-generated deepfakes were used to mislead and manipulate public opinion, prompting a crackdown by the National Election Commission.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The incident involves the coordinated use of AI-generated deepfakes to influence political processes and public opinion during democratic elections.

Additional risk subdomains

• 6.5 Governance failure: The report highlights that the rapid speed of deepfake production outpaces the verification capabilities of election regulatory bodies.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The risk is caused by human actors intentionally deploying trained AI deepfake generation models to create misleading media for political campaigning.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the use of AI-generated deepfake videos and images, which fall under the category of AI-generated content requiring transparency obligations.

AI system and alleged parties

• AI system: None named
• AI purpose: Deepfake Video Generation; Deepfake Image Generation
• Behaviour type: Tool
• Alleged developer: Unknown political operatives, Unknown political groups
• Alleged deployer: Unknown political operatives, Unknown political groups
• Alleged harmed parties: Yoon Suk-yeol, Korean voters, Journalism, Electoral integrity, Democracy

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Minor, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes

Directly caused: 129 pieces of AI-generated media content were detected as violating election laws.

Inferred additional harm: It is highly likely that more than the 129 detected deepfakes were created and circulated across social media platforms, exposing a large portion of the electorate to malicious synthetic media.

Democracy

Reported: Yes

Directly caused: The integrity of the South Korean parliamentary elections was threatened by 129 detected instances of illegal deepfake campaigning.

Indirectly caused: Past incidents include a deepfake of Yoon Suk Yeol endorsing a candidate and a US robocall advising voters not to vote, directly interfering with democratic participation.

Inferred additional harm: Continued proliferation of undetected deepfakes could lead to widespread erosion of public trust in democratic institutions and election outcomes.

Epistemic

Reported: Yes

Directly caused: The 129 detected deepfakes spread false or misleading representations of candidates and political situations.

Inferred additional harm: The rapid generation of believable fakes outpaces verification, potentially leading to a polluted information ecosystem where voters cannot distinguish real from synthetic media.

People affected

• Occurrences reported: 129

Potential causes

Technology

• Rapid Deepfake Production: Deepfake creation speed outpaces election authority verification efforts.
• Evolving Deepfake Sophistication: Constantly evolving AI technology poses ongoing hurdles for detection.

Human Factors

• Creation of Misleading Media: Actors generate fake videos to manipulate voters and mislead public opinion.

Process and Methods

• Lagging Verification Methods: Verification protocols cannot match the speed of deepfake dissemination.

Regulatory Environment

• Recent Law Enforcement Gap: The ban on deepfakes was only recently implemented, leaving legacy issues.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the context of the South Korean elections, the specific number of detected deepfakes (129), and the regulatory response. The role of AI as a tool for generating deceptive media is explicit, and the primary risk to democratic norms is well-documented.
• Ambiguities identified: The specific AI models used to generate the deepfakes are not identified, and the exact content or target of the 129 detected deepfakes is not detailed.
• Alternative interpretations: None. The incident is clearly a case of malicious use of synthetic media to influence elections.

During the 2024 South Korean legislative election cycle, authorities detected 129 illegal AI-generated deepfakes used for campaigning. The incident highlights the growing threat of synthetic media to electoral integrity and democratic sovereignty, as the rapid generation of deceptive content outpaces the verification capabilities of regulatory bodies.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: South Korea
• Other affected: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Near-term. The threat was active during the election campaign period, requiring continuous monitoring and response from election authorities.
• Autonomy: Human-controlled. AI models were used as tools by human actors who generated and distributed the deepfakes.
• Novelty: Evolved capability. While deepfakes have been used before, the volume and speed of production outpacing regulatory verification capabilities represent an evolved threat.

Impact by dimension

• Physical security: Negligible. No physical systems, kinetic threats, or critical infrastructure were targeted or affected in this incident.
• Information security: Substantial. The systematic use of 129 AI-generated deepfakes represents a notable information warfare effort to manipulate public opinion and spread disinformation during a major national election.
• Sovereignty: Substantial. The integrity of democratic electoral processes and legislative elections was threatened by illegal deepfake campaigning, challenging core government functions.
• Economic security: Negligible. There were no reported threats to financial systems, strategic technologies, or economic stability.
• Societal stability: Minor. While designed to manipulate voters, the deepfakes caused limited immediate disruption to overall societal stability and did not lead to widespread civil unrest.