OpenAI identified and banned two clusters of China-linked ChatGPT accounts that were used to conduct covert influence operations targeting U.S. AI policy debates. The accounts generated social media comments, images, and false claims about data centers and U.S. tech competition to amplify domestic divisions. While the operations sought to manipulate public opinion, OpenAI found no evidence of significant breakout beyond the accounts' own activity.
OpenAI reported that two clusters of ChatGPT accounts likely originating from China used its models to support apparent covert influence operations about U.S. AI and technology policy. The accounts reportedly generated a variety of media and social media comments about AI data centers, electricity costs, tariffs, and U.S. tech competition before OpenAI banned them.
Risk classification
- Primary risk domain: 4 Malicious actors
- Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale
The incident involved foreign state-linked actors using ChatGPT to generate coordinated, covert influence campaigns aimed at manipulating public opinion and political debates regarding U.S. AI infrastructure.
Additional risk subdomains
- 3.1 False or misleading information: The campaigns generated false claims, such as alleging ChatGPT user data had been compromised, to mislead the public.
Causal factors
- Entity: Human
- Intent: Intentional
- Timing: Post-deployment
The risk was caused by human threat actors intentionally using deployed ChatGPT models to generate and spread covert influence campaign content.
EU AI Act risk tier
- Risk tier: 3 Limited Risk
Limited Risk: The incident involves the use of 'ChatGPT' to generate 'social media comments and images', which falls under the category of chatbots and AI-generated content requiring transparency obligations.
AI system and alleged parties
- AI system: ChatGPT (OpenAI)
- AI purpose: Social Media Content Generation; Writing Assistant
- Behaviour type: Assistant
- Alleged developer: Synthetic media generation technology developers, OpenAI, Large language model developers, Image generation technology developers, Chatbot developers
- Alleged deployer: People's Republic of China-linked influence operators, OpenAI users, Information manipulation actors in China, Information manipulation actors, ChatGPT users
- Alleged harmed parties: Social media users, People participating in U.S. AI policy debates, OpenAI, National security and intelligence stakeholders, General public of the United Statese, General public, Epistemic integrity, Communities debating AI data center development
Harm severity
Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.
- Physical: direct Negligible, indirect Negligible
- Infrastructure: direct Negligible, indirect Negligible
- Property: direct Negligible, indirect Negligible
- Financial: direct Negligible, indirect Negligible
- Environmental: direct Negligible, indirect Negligible
- Malicious content: direct Minor, indirect Negligible
- Differential treatment: direct Negligible, indirect Negligible
- Civil rights: direct Negligible, indirect Negligible
- Democracy: direct Minor, indirect Negligible
- Privacy: direct Negligible, indirect Negligible
- Psychological: direct Negligible, indirect Negligible
- Epistemic: direct Minor, indirect Negligible
- Child sexual exploitation and abuse: direct Negligible, indirect Negligible
Malicious content
Reported: Yes, the report describes the generation of covert influence campaign content, including social media comments and images designed to stoke division.
Directly caused: ChatGPT was used to generate comments and images claiming data centers increase electricity prices and criticizing US tariffs.
Indirectly caused: N/A
Inferred additional harm: It is likely that additional undetected spam or low-quality political comments were generated using similar AI tools.
Democracy
Reported: Yes, the report describes attempts by foreign actors to covertly interfere in democratic societies and manipulate public debates.
Directly caused: The campaigns targeted US AI policy debates and sought to stoke domestic divisions regarding energy and infrastructure.
Indirectly caused: N/A
Inferred additional harm: N/A
Epistemic
Reported: Yes, the report describes the generation of false claims, such as alleging ChatGPT user data had been compromised.
Directly caused: The second cluster generated false allegations claiming ChatGPT user data had been compromised.
Indirectly caused: N/A
Inferred additional harm: N/A
People affected
Potential causes
Management
- Reactive Safety Measures: AI developers ban accounts only after influence campaigns have run.
- Siloed Threat Intelligence: Security, PR, and government teams fail to share a unified threat picture.
- Engagement-Driven Business Model: Social platforms prioritize user engagement over narrative integrity.
Technology
- Generative AI Abuse: ChatGPT used to easily generate covert propaganda comments and comic strips.
- AI-Narrated Content Farms: Machine voiceovers and automated templates scale up fake news delivery.
- Algorithmic Amplification: Social media algorithms push AI slop to targeted local audiences.
Data Inputs
- Inauthentic Social Data: Networks of fake profiles seed AI-generated imagery into local groups.
- Fabricated Tech Articles: Fake articles hosted on domains imitating real tech news outlets.
- Exploited Satellite Imagery: Real satellite images used to anchor false claims about infrastructure.
Human Factors
- Pre-existing Public Anxiety: Existing fears about AI, grid stability, and jobs are easily exploited.
- Low Digital Literacy: Users fail to distinguish AI-generated propaganda from authentic news.
- Hyperlocal Grievances: Local opposition to data centers makes residents receptive to fake claims.
Process and Methods
- Delayed Threat Detection: Inability to detect coordinated AI operations before they spread widely.
- Dismantled Tracking Teams: Reduction of government teams tracking foreign influence operations.
- Replicable Propaganda Aesthetics: Domestic actors easily copy adversarial AI Lego-style video templates.
Regulatory Environment
- Weak AI Governance: Absence of clear international rules governing AI-driven influence.
- Geopolitical Friction: Adversarial states use AI to covertly weaken democratic institutions.
- Lack of Platform Regulation: No regulatory penalties for hosting coordinated inauthentic AI content.
Information quality
- Classification confidence: High
- Reason for confidence: The reports from OpenAI and Alethea provide clear, detailed, and consistent accounts of the threat actors, their tactics, the specific campaigns, and the lack of real-world impact. There is high consensus on the facts of the ChatGPT misuse.
- Ambiguities identified: None of significance; the scale of the campaigns' actual reach is slightly vague, but the core facts of the AI's role are clear.
China-linked actors used OpenAI's ChatGPT to conduct covert influence operations targeting U.S. AI policy debates. The operations generated social media content to exploit domestic controversies but were detected and banned by OpenAI, achieving little to no authentic engagement and presenting minor national security impact.
- Overall national security impact: Minor
- Response level: Moderate
- Scope: Single nation
- Primary target: United States
- Alleged perpetrator: China
Threat characteristics
- Imminence: Long-term. The immediate threat was mitigated by banning the accounts, representing an ongoing strategic concern rather than an active crisis.
- Autonomy: Human-controlled. The AI system was used as a tool to assist human operators who provided direct prompts and controlled the dissemination of content.
- Novelty: Evolved capability. Represents an evolution of traditional covert influence operations by integrating generative AI models to scale content creation.
Impact by dimension
- Physical security: Negligible. No physical systems, critical infrastructure, or human safety threats were targeted or affected in this incident.
- Information security: Minor. State-linked threat actors utilized generative AI to scale influence campaigns, but the effort was successfully mitigated with little to no authentic engagement.
- Sovereignty: Minor. Foreign actors attempted to covertly influence U.S. domestic policy debates, though the operation failed to disrupt any government functions or decisions.
- Economic security: Negligible. No financial systems or strategic technologies were compromised, resulting in negligible economic or technological security impact.
- Societal stability: Minor. The operation attempted to stoke domestic division over energy and tariffs but failed to generate public traction or civil unrest.
.avif){kind=small}