Covert AI Influence Operations Linked to Russia, China, Iran, and Israel, OpenAI Reports

In a report released by OpenAI, the company described how its generative AI tools were misused by state actors and private companies in Russia, China, Iran, and Israel to conduct covert influence campaigns aimed at manipulating public opinion and geopolitical narratives.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The incident involved multiple state-sponsored and commercial actors using generative AI to run coordinated, large-scale covert influence operations targeting political processes and public opinion globally.

Additional risk subdomains

• 3.1 False or misleading information: The threat actors used AI to generate false or misleading articles, headlines, and social media comments to deceive the public.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The risk was caused by human threat actors who intentionally used deployed AI models to run covert influence operations.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes chatbots and AI-generated content used in influence campaigns. According to the definitions, Risk Level 3 includes 'Chatbots' and 'AI-generated content such as deepfakes' which require transparency obligations.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Social Media Content Generation; Translation
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: Zeno Zeno, Spamouflage, Russian government, Israeli government, Iranian government, International Union of Virtual Media, Doppelganger, Chinese government
• Alleged harmed parties: United States, Ukraine, Truth, social media users, National security and intelligence stakeholders, Moldova, Lithuania, Latvia, General public of the United States, General public of the European Union, General public, Estonia, Epistemic integrity, Electoral integrity, Democracy, Critics of the Chinese government

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Minor, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports describe the creation and spread of deceptive political content, anti-Ukraine comments, and anti-Islamic messages.

Directly caused: OpenAI's models were used to directly generate anti-Ukraine comments, anti-Islamic messages, and disparaging posts targeting Chinese dissidents across platforms like X, Telegram, and Facebook.

Inferred additional harm: It is likely that additional undetected or unmentioned malicious content was generated by these or similar campaigns before disruption.

Democracy

Reported: Yes, the reports describe campaigns designed to deceptively manipulate public opinion, influence geopolitics, and target democratic elections.

Directly caused: The campaigns used AI to generate political content, fake biographies, and articles aiming to influence elections in the US, Europe, and India, and to sow internal discord.

Inferred additional harm: While OpenAI noted these campaigns had low engagement, the cumulative effect of multiple state-sponsored operations represents an ongoing threat to public trust in democratic institutions.

Epistemic

Reported: Yes, the reports describe the generation of fake news, fictional personas, and deceptive articles to manipulate public opinion.

Directly caused: AI was used to generate fictional personas, fake biographies, and deceptive news articles on websites like iuvmpress[.]co and revealscum[.]com.

Inferred additional harm: The generation of synthetic deceptive content contributes to the broader erosion of trust in online information, even if individual campaigns had low reach.

People affected

• Occurrences reported: 5

Potential causes

Management

• Difficulties in Distribution Tracking: AI developers struggled to track how generated content was distributed.
• Incomplete Threat Sharing: Historic gaps in industry-wide sharing of threat indicators.

Technology

• Abuse of Generative AI Tools: Threat actors exploited LLMs to generate deceptive content at scale.
• AI-Assisted Code Debugging: Actors used models to debug bot code and automate distribution.
• Model Vulnerability to Exploitation: Models generated persuasive text and translations for foreign campaigns.

Data Inputs

• Social Media Query Data: Actors queried models with social media data to optimize campaigns.
• Research of Current Events: Threat actors used AI to research current events for propaganda.

Human Factors

• Malicious Actor Intent: State-sponsored actors actively sought to manipulate public opinion.
• Operator Human Error: Actors accidentally published model refusal messages on social media.
• Fictional Persona Creation: Humans used AI to generate fake biographies to mimic real users.

Process and Methods

• Multi-Platform Distribution: Lack of coordination between generation tools and distribution platforms.
• Mixing AI and Manual Content: Combining AI text with memes made campaigns harder to detect initially.
• Deceptive Article Translation: Using AI to translate and edit articles to bypass language barriers.

Regulatory Environment

• Lack of Global AI Governance: Absence of international laws allowed state actors to exploit AI tools.
• Sanction Evasion by State Actors: Sanctioned groups like Iranian Guards accessed US-based AI technologies.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from both the New York Times and OpenAI provide highly detailed, consistent, and first-hand accounts of the disrupted operations, including their names, origins, methods, and lack of audience engagement.
• Ambiguities identified: The exact number of views or impressions the campaigns received before disruption is not quantified, described only as having 'few replies and likes' and 'struggling to build an audience'.

Disruption of five state-sponsored covert influence operations from Russia, China, Iran, and Israel utilizing generative AI to manipulate public opinion and target global elections. While the campaigns achieved low engagement and were successfully disrupted, they demonstrate an evolving information warfare threat where adversaries leverage AI to scale deceptive content generation.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: Multiple democratic nations
• Other affected: United States, European Union, India
• Alleged perpetrator: State-sponsored groups from Russia, China, Iran, and Israel

Threat characteristics

• Imminence: Long-term. Represents an ongoing, strategic capability concern as state actors continuously refine AI-enabled disinformation tactics.
• Autonomy: Human-controlled. Human operators orchestrated the campaigns, using AI as a tool to generate content and increase efficiency under human direction.
• Novelty: Evolved capability. Represents an evolution of traditional covert influence operations, leveraging generative AI to scale content creation and reduce language errors.

Impact by dimension

• Physical security: Negligible. No physical security threats, critical infrastructure compromise, or kinetic actions were reported in connection with these operations.
• Information security: Substantial. Multiple state-sponsored actors utilized generative AI to scale covert influence operations and manipulate geopolitical narratives, representing a substantial information warfare threat.
• Sovereignty: Minor. The campaigns targeted democratic elections and political processes globally, but failed to achieve significant reach or disrupt government functions.
• Economic security: Negligible. There was no reported threat to critical economic systems, strategic industries, or technological infrastructure.
• Societal stability: Minor. The campaigns attempted to manipulate public opinion and sow discord on sensitive topics, but had minimal impact on societal stability due to low engagement.