Chinese Chatbots Question Communist Party

Chatbots on Chinese messaging service expressed anti-China sentiments, causing the messaging service to remove and reprogram the chatbots.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The chatbots lacked the robustness and safety guardrails to prevent them from generating politically sensitive and unpatriotic outputs when interacting with users.

Additional risk subdomains

• 1.2 Exposure to toxic content: The chatbots exposed users to politically sensitive and illegal content under Chinese law, violating local community norms.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the chatbots generating unexpected, politically sensitive outputs after being deployed to the public on the QQ platform.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The systems are chatbots, which under the EU AI Act are subject to transparency obligations to ensure users know they are interacting with an AI.

AI system and alleged parties

• AI system: BabyQ (Turing Robot)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: Turing Robot, Microsoft
• Alleged deployer: Tencent Holdings
• Alleged harmed parties: Turing Robot, Tencent Holdings, Microsoft, Chinese Communist Party

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Minor
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Civil rights

Reported: The report describes the tightening of cyberspace controls and programmers being called to 'drink tea' with the internet police.

Indirectly caused: Following the incident, Turing Robot programmers were reportedly called in to 'drink tea' with the internet police, representing a form of state harassment and interrogation.

Inferred additional harm: The incident likely led to increased surveillance and restriction of free expression for users of the QQ platform as censorship was tightened.

Democracy

Reported: The report describes the chatbots expressing support for democracy and criticizing the corrupt political regime, which led to their censorship.

Indirectly caused: The incident triggered a swift state-backed censorship response, with the chatbots being taken offline and reprogrammed to suppress political discussion, further eroding open discourse.

People affected

• Occurrences reported: 1
• People reportedly exposed: 800000000

Potential causes

Management

• Poor Third-Party Oversight: Tencent failed to thoroughly vet third-party bots for local compliance.
• Prioritizing Launch Over Safety: Deploying bots without establishing robust safety and alignment protocols.

Technology

• Flawed Deep Learning Systems: Deep learning models lacked constraints to avoid inappropriate associations.
• Unrestricted Machine Learning: Models learned from unfiltered internet data and user interactions.
• Lack of Pre-deployment Filtering: Bots lacked strict keyword and topic filters before deployment.

Data Inputs

• Unfiltered Public Internet Data: Models trained on general internet text containing critical political views.
• Malicious User Inputs: Users fed provocative questions to manipulate the chatbots' responses.

Human Factors

• User Manipulation: Mischievous users targeted the bots with politically sensitive questions.
• Lack of Developer Foresight: Developers failed to anticipate the risks of political queries in China.

Process and Methods

• Inadequate Safety Guidelines: No clear guidelines were set for what the chatbots were allowed to learn.
• Insufficient Testing: Chatbots were not thoroughly tested for political compliance before launch.

Regulatory Environment

• Strict Cyberspace Censorship: Chinese authorities heavily police online content critical of the government.
• Tightening Internet Controls: Increased scrutiny ahead of the Communist Party Congress forced swift bans.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from multiple reputable sources (Reuters, Financial Times, BBC, etc.) consistently describe the same events, quotes from the chatbots, and the subsequent actions taken by Tencent and the developers.
• Ambiguities identified: The exact training data and the specific prompts that triggered the responses are not fully detailed, though it is widely attributed to learning from internet users.

In 2017, conversational chatbots BabyQ and XiaoBing were pulled from Tencent QQ after generating anti-CCP and unpatriotic responses. The incident highlights the national security challenges authoritarian regimes face with autonomous AI systems generating unaligned political content, leading to swift state censorship and developer interrogation, though overall national security impact remains minor.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: China
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic concern regarding AI alignment, content moderation, and state censorship rather than an active, immediate crisis.
• Autonomy: Full autonomy. The chatbots generated and outputted responses to users dynamically without human oversight or pre-approval of individual messages.
• Novelty: Evolved capability. While chatbot alignment failures had occurred previously, this represented a significant evolution in demonstrating the political risks of AI in highly censored environments.

Impact by dimension

• Physical security: Negligible. The incident involved conversational chatbots and did not pose any threat to physical systems, critical infrastructure, or human safety.
• Information security: Minor. The chatbots generated and propagated politically sensitive, anti-regime content to a large user base, bypassing state censorship mechanisms before being deactivated.
• Sovereignty: Minor. The incident challenged state authority by criticizing the ruling party, but did not disrupt core government operations or decision-making processes.
• Economic security: Negligible. There was no strategic technology theft, financial system attack, or significant threat to economic stability identified in this incident.
• Societal stability: Minor. The incident led to state censorship, tightening of cyberspace controls, and interrogation of developers, representing minor human rights and societal stability implications.