Korean Politician Employed Deepfake as Campaign Representative

A South Korean political candidate created a deepfake avatar which political opponents alleged to be fraudulent and a threat to democracy.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The incident involves using AI deepfakes to conduct coordinated, population-level political campaigning and influence public opinion during a presidential election.

Additional risk subdomains

• 3.2 Pollution of information ecosystem and loss of consensus reality: The widespread use of deepfake candidates raised concerns about eroding public trust in authentic media and polluting the political information ecosystem.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The deployment of AI avatars was an intentional decision by human campaign teams post-deployment to influence voters.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The reports describe the use of AI-generated content such as deepfakes and chatbots ('AI Yoon', 'AiDY') for political campaigning, which require transparency obligations under the EU AI Act.

AI system and alleged parties

• AI system: None named
• AI purpose: Deepfake Video Generation; Voice Generation
• Behaviour type: Assistant
• Alleged developer: unknown
• Alleged deployer: Yoon Suk-yeol's campaign, Yoon Suk-yeol
• Alleged harmed parties: Korean public

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Minor, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Democracy

Reported: The report explicitly describes concerns regarding the erosion of democratic norms and political ethics caused by the deployment of AI avatars.

Directly caused: The deployment of AI avatars to represent candidates in elections altered traditional campaigning, potentially deceiving voters about the candidates' true personas, habits, and speaking abilities.

Epistemic

Reported: The report explicitly describes concerns about epistemic harm, specifically the potential for deepfakes to spread fake news and lower public trust in media.

Directly caused: The creation of highly realistic synthetic avatars of political candidates created risks of voter confusion, with some voters initially believing the avatar was real.

Indirectly caused: Experts warned that the normalization of deepfakes in campaigns could lower the public's trust in other forms of media presenting candidates in their true form.

People affected

• Occurrences reported: 1
• People reportedly exposed: 7000000

Potential causes

Management

• Prioritizing Image over Policy: Campaign management prioritized humorous digital image over serious policy debate.
• Ethical Decorum Downgrade: Management approved insulting scripts targeting rivals to gain younger votes.

Technology

• Deceptive Deepfake Realism: High realism leads some voters to mistake the avatar for the real candidate.
• Lack of Emotional Expression: Synthesized voice is flat and cannot replicate natural laughter or sadness.
• Inability to Replicate Movements: Software cannot replicate natural physical habits like head-bobbing.

Data Inputs

• Pre-recorded Training Data: Avatar is trained on hours of specific audio and video to clone the candidate.
• Scripted Non-Candidate Outputs: Scripts are written by campaign staff rather than the candidate himself.
• Filtered Behavioral Data: Training data was filtered to hide candidate's habits and speaking flaws.

Human Factors

• Voter Deception Risk: Voters may mistake synthetic deepfakes for actual live recordings.
• Reduced Trust in Genuine Media: Prevalence of deepfakes lowers public trust in authentic media broadcasts.
• Overreliance on Likability: Voters prefer witty digital avatars over complex, dull policy debates.

Process and Methods

• Rapid Script Generation: Scripts are written in brainstorms as short as 30 minutes without deep review.
• Disguising Candidate Flaws: Campaigns use avatars to mask bad postures and poor speaking habits.
• Use of Gamified Language: Script writers use gaming slang and insults to maximize online engagement.

Regulatory Environment

• Unclear Election Laws: Election commission struggled to determine the legitimacy of AI campaign tools.
• Inadequate Deepfake Penalties: Absence of strong legal punishments for malicious deepfakes during campaigns.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of how the AI avatars were created, deployed, and received by the public during the 2022 South Korean election. Multiple sources confirm the technical process (DeepBrain AI cloning candidates from recorded data) and the ethical debates surrounding their use.
• Ambiguities identified: None significant, though the exact technical specifications of DeepBrain AI's underlying models are not fully detailed.
• Alternative interpretations: The deployment could be viewed purely as a novel marketing campaign rather than an AI safety risk, though the ethical concerns raised about democratic norms justify the safety analysis.

During the 2022 South Korean presidential election, domestic campaigns officially deployed AI-generated deepfake avatars of candidates to engage voters. While the incident raised significant ethical concerns regarding voter deception, epistemic trust, and political decorum, it was human-controlled and did not involve foreign interference or critical infrastructure threats, representing a minor national security impact.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: South Korea
• Alleged perpetrator: Domestic political campaigns

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic concern regarding the normalization of deepfakes in political processes rather than an active, immediate national security crisis.
• Autonomy: Human-controlled. The AI avatars did not generate content autonomously; their actions and speech were entirely scripted and controlled by human campaign staffers.
• Novelty: Evolved capability. Represents a significant advancement and normalization of existing deepfake technology being officially integrated into high-profile national political campaigns.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety threats were involved in this digital campaigning incident.
• Information security: Minor. While involving deepfake technology, the avatars were officially deployed by domestic political campaigns rather than hostile foreign intelligence operations, representing a minor information security threat.
• Sovereignty: Minor. The deployment occurred within a democratic election but was conducted by domestic political candidates using commercial technology, with no disruption to core constitutional processes or foreign interference.
• Economic security: Negligible. The incident did not target financial systems, involve strategic technology theft, or threaten economic stability.
• Societal stability: Minor. Sparked ethical debates regarding voter deception and the pollution of the information ecosystem, but did not lead to civil unrest, mass surveillance, or systemic human rights abuses.