AI-Powered Chinese Surveillance Campaign 'Peer Review' Used for Real-Time Monitoring of Anti-State Speech on Western Social Media

OpenAI reportedly uncovered evidence of a Chinese state-linked AI-powered surveillance campaign, dubbed "Peer Review," designed to monitor and report anti-state speech on Western social media in real time. The system, believed to be built on Meta’s open-source Llama model, was detected when a developer allegedly used OpenAI’s technology to debug its code. OpenAI also reportedly identified disinformation efforts targeting Chinese dissidents and spreading propaganda in Latin America.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.1 Disinformation, surveillance, and influence at scale

The incident involves coordinated, state-linked campaigns using AI to develop surveillance tools for monitoring public protests and to generate large-scale political disinformation.

Additional risk subdomains

• 4.3 Fraud, scams, and targeted manipulation: The reports also detail Cambodia-based romance and task scams, as well as North Korean employment fraud, which target specific individuals for financial gain.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human threat actors who intentionally leveraged deployed AI models to facilitate surveillance, disinformation, and financial scams.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes the misuse of ChatGPT, which is a chatbot, and the generation of AI-generated content for disinformation and scams, both of which fall under the Limited Risk category requiring transparency obligations.

AI system and alleged parties

• AI system: ChatGPT, Llama 3.1 (Meta, OpenAI)
• AI purpose: Writing Assistant; Code Generation
• Behaviour type: Assistant
• Alleged developer: Various open-source AI developers, OpenAI, Meta, Chinese state security researchers
• Alleged deployer: Chinese state-linked actors, Chinese Communist Party
• Alleged harmed parties: Western social media communities, Social media users in Latin America, social media users, Opposition voices against the Chinese Communist Party, Chinese dissidents, Cai Xia

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Minor, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports describe the generation of disinformation, critical posts targeting dissidents, and deceptive comments for scams.

Directly caused: ChatGPT was used to generate English-language comments attacking dissident Cai Xia, Spanish-language news articles critical of the U.S., and deceptive comments for romance and task scams.

Inferred additional harm: It is likely that a large volume of malicious content reached thousands of social media users and news readers across Latin America and other regions.

Civil rights

Reported: Yes, the reports describe attempts to build a surveillance tool to monitor anti-China protests and share insights with authorities.

Directly caused: The 'Peer Review' campaign used AI to debug a tool designed to monitor anti-Chinese sentiment and spot calls for protests, with the intent of sharing this with Chinese authorities, threatening freedom of speech and assembly.

Inferred additional harm: If the surveillance tool was successfully deployed, it likely led to the violation of civil rights and potential persecution of numerous individuals protesting against the Chinese government.

Democracy

Reported: Yes, the reports describe coordinated influence campaigns targeting public opinion and elections.

Directly caused: Threat actors used AI to generate articles critical of the US published in Latin American media, and to target the Ghanaian presidential election with covert influence operations.

Inferred additional harm: These coordinated campaigns likely eroded trust in democratic institutions and manipulated public opinion in the targeted countries.

Privacy

Reported: Yes, the reports describe the development of a social media surveillance tool to monitor individuals.

Directly caused: The 'Peer Review' campaign attempted to build a tool to collect real-time data and reports about protests and anti-China sentiment on platforms like X, Facebook, and YouTube.

Inferred additional harm: The surveillance tool likely monitored and analyzed the public posts of thousands of social media users without their explicit consent, infringing on their privacy expectations.

Epistemic

Reported: Yes, the reports describe the generation and translation of false or misleading articles and social media comments.

Directly caused: AI was used to translate and generate long-form articles critical of the U.S. and publish them on Latin American news sites, as well as generate fake resumes and profiles for North Korean deceptive employment schemes.

Inferred additional harm: The proliferation of AI-generated disinformation and fake professional profiles likely degraded the overall information ecosystem and trust in online platforms.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Policy Enforcement Gaps: OpenAI's policy bans surveillance, but enforcement relied on post-hoc detection.

Technology

• Open-Source Model Accessibility: Llama models are open-source, allowing local deployment without oversight.
• Code Debugging Capabilities: ChatGPT's coding abilities allowed actors to debug surveillance spyware.
• Language Translation Power: LLMs easily translated disinformation articles into Spanish to spread influence.

Data Inputs

• Social Media Data Ingestion: Surveillance tool ingested real-time posts from X, Facebook, and Instagram.
• Document Screenshots Analysis: ChatGPT analyzed screenshots of English documents showing protest announcements.
• Public Information Scraping: ChatGPT researched public data on Western politicians and think tanks.

Human Factors

• Manual Prompting Techniques: Actors used manual prompting to mimic normal users and evade detection.
• Malicious Intent of State Actors: Chinese operators actively sought to monitor dissidents and anti-China protests.
• Deceptive Social Engineering: Fraudsters used AI to craft highly convincing romance scams to trick victims.

Process and Methods

• Model Misuse for Spyware: Threat actors bypassed standard use cases to debug surveillance tool code.
• Local Deployment of Models: Running open-source models locally prevents external monitoring of activities.
• Fragmented AI Tool Usage: Using multiple models (Llama, DeepSeek, ChatGPT) obscured the threat signal.

Regulatory Environment

• Lack of Open-Source Controls: No regulations restrict the distribution or local hosting of powerful LLMs.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from multiple reputable sources consistently describe the same set of campaigns, their origins, and how OpenAI detected and mitigated them. The details regarding the 'Peer Review' and 'Sponsored Discontent' campaigns are clear and corroborated across all sources.
• Ambiguities identified: The exact extent of the deployment of the 'Peer Review' surveillance tool and the specific number of victims affected by the Cambodian scams are not fully quantified.
• Alternative interpretations: The activities could be viewed purely as traditional cyber espionage and influence operations that incidentally used AI, rather than failures of AI safety itself.

State-linked and criminal actors leveraged AI tools to enhance disinformation, develop social media surveillance capabilities, and conduct financial scams. While the AI acted primarily as a human-controlled assistant, the integration of LLMs into state-sponsored intelligence and influence operations represents a substantial, evolving threat to information security and democratic stability.

• Overall national security impact: Substantial
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, China, Ghana, Latin America, Cambodia, North Korea, Iran
• Alleged perpetrator: State-linked actors from China, North Korea, and Iran, and cybercriminal networks in Cambodia

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic and capability concern involving multiple state-sponsored and criminal actors rather than an immediate, singular crisis.
• Autonomy: Human-controlled. The AI systems (ChatGPT and Llama) were used as assistants for coding, translation, and text generation, with human actors directing and executing the campaigns.
• Novelty: Evolved capability. Represents a significant advancement where threat actors use LLMs to scale and refine established methods of espionage, disinformation, and fraud.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure compromise were reported in connection with these campaigns.
• Information security: Substantial. State-linked actors from China, Iran, and North Korea used AI to scale disinformation, generate anti-US propaganda, and develop social media surveillance tools, representing notable information warfare operations.
• Sovereignty: Substantial. Covert influence campaigns targeted the Ghanaian presidential election and public opinion in democratic nations, seeking to manipulate political discourse and undermine government functions.
• Economic security: Minor. North Korean actors used AI to generate fake resumes for deceptive employment schemes, and Cambodian groups conducted financial scams, presenting minor threats to economic security.
• Societal stability: Substantial. The Chinese 'Peer Review' campaign aimed to build surveillance tools monitoring anti-state protests, directly threatening human rights, privacy, and freedom of speech.