North Korea's Kimsuky Group Reportedly Uses AI-Generated Military ID Deepfakes in Phishing Campaign

Genians reported a phishing campaign by North Korea's Kimsuky group using purportedly AI-generated deepfake military ID cards. Emails reportedly impersonating South Korean defense institutions carried ZIP files with forged IDs whose photos were reportedly created using generative AI. When opened, hidden malware reportedly executed, downloading scripts disguised as Hancom Office updates. This reportedly marked an evolution in Kimsuky's tactics, using AI decoys to boost social engineering.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The Kimsuky group used ChatGPT to generate a fake military ID to impersonate a military official and deceive targeted individuals in a spear-phishing campaign, which constitutes targeted exploitation and impersonation for malicious gain.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI system failed to maintain its safety guardrails, allowing the attackers to bypass its refusal filters and generate illegal counterfeit identification documents.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human threat actors (Kimsuky) intentionally abusing a deployed AI system (ChatGPT) to generate deceptive assets for a cyberattack.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the use of AI to generate deepfakes (fake military ID cards). Under the EU AI Act, AI systems that generate or manipulate image, audio or video content (deepfakes) pose a moderate risk and are subject to specific transparency obligations.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Deepfake Image Generation; Writing Assistant
• Behaviour type: Tool
• Alleged developer: OpenAI
• Alleged deployer: Velvet Chollima, THALLIUM, Reconnaissance General Bureau, Kimsuky Group, Group 0094, Government of North Korea, Emerald Sleet, Black Banshee, APT43
• Alleged harmed parties: Truth, South Korean defense personnel, National security and intelligence stakeholders, Government of South Korea, General public of South Korea, Epistemic integrity

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Minor
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report describes a phishing campaign designed for data theft, but does not explicitly detail specific successful data leaks.

Indirectly caused: The malware delivered via the phishing campaign was designed to enable internal data theft and remote control of victims' devices.

Inferred additional harm: It is highly likely that some targeted individuals had their sensitive personal data, credentials, or research materials compromised as a result of the campaign.

People affected

• Occurrences reported: 1
• People reportedly exposed: 50

Potential causes

Management

• Inadequate AI Risk Monitoring: Organizations lacked proactive monitoring for AI misuse in operations.
• Weak Endpoint Security Adoption: Lack of EDR systems to detect obfuscated scripts triggered by AI phishing.

Technology

• Bypassing Model Safeguards: Attackers bypassed ChatGPT safeguards using mock-up framing prompts.
• Deepfake Generation Ease: Generative AI makes creating highly convincing fake military IDs straightforward.
• AI-Assisted Identity Manipulation: AI tools enable North Korean workers to generate fake virtual identities.

Data Inputs

• Adversarial Prompt Framing: Manipulated prompt inputs bypassed the model's refusal triggers.
• Public ID Data Availability: Publicly available military ID images on the internet were used as reference.

Human Factors

• Trust in AI Realism: Victims trusted highly realistic AI-generated ID images in emails.
• Limited Worker Programming Skills: North Korean workers relied on AI to compensate for poor coding skills.
• Language Communication Gaps: Workers used AI to overcome restricted English-based communication skills.

Process and Methods

• Insufficient Recruitment Screening: Technical assessment processes failed to detect AI-assisted candidate fraud.
• Inadequate Phishing Detection: Traditional email security failed to flag AI-generated deepfake decoys.

Regulatory Environment

• Sanction Circumvention via AI: State-sponsored actors leveraged AI to bypass international trade sanctions.
• Lack of Deepfake Controls: Absence of strict global regulations on generating official document images.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from Genians provide highly detailed technical analysis of the phishing campaign, including the specific malware chain, C2 servers, and the role of ChatGPT in generating the fake ID. The metadata of the PNG file explicitly confirmed ChatGPT's involvement, leaving little ambiguity about the AI's role.
• Ambiguities identified: The exact number of successfully compromised victims and the specific prompts used to bypass ChatGPT's safety filters are not fully detailed.
• Alternative interpretations: None. The incident is clearly a state-sponsored cyber-espionage campaign leveraging AI-generated social engineering decoys.

North Korean state-affiliated group Kimsuky utilized ChatGPT to generate highly realistic fake South Korean military ID cards to enhance a spear-phishing campaign targeting defense and research sectors. This incident highlights the growing threat of generative AI being leveraged by state actors to bypass safety filters and produce sophisticated social engineering assets for cyber-espionage.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: South Korea
• Alleged perpetrator: Kimsuky

Threat characteristics

• Imminence: Long-term. Represents an evolution in threat actor tactics and ongoing cyber-espionage capabilities rather than an active, immediate physical crisis.
• Autonomy: Human-controlled. The AI system (ChatGPT) was strictly used as a tool by human operators to generate static images, with no autonomous decision-making or execution.
• Novelty: Evolved capability. Demonstrates an evolved capability where threat actors successfully bypass commercial AI safety guardrails to generate highly realistic counterfeit government credentials.

Impact by dimension

• Physical security: Negligible. No physical systems, kinetic attacks, or critical infrastructure control systems were compromised in this cyber-espionage incident.
• Information security: Substantial. State-sponsored threat actor Kimsuky used AI-generated fake military IDs to conduct high-quality spear-phishing against defense and research sectors, aimed at intelligence gathering and data theft.
• Sovereignty: Minor. The campaign targeted South Korean defense-related institutions, but there is no evidence of direct compromise to core government decision-making or constitutional processes.
• Economic security: Minor. Potential compromise of research data in defense and North Korean studies, representing a minor threat to technological and strategic intelligence security.
• Societal stability: Minor. Targeting of human rights activists and journalists poses a risk to individual civil liberties, but does not represent a population-scale threat to societal stability.