WeChat’s Machine Translation Gave a Racist English Translation for the Chinese Term for “Black Foreigner”

The Chinese platform WeChat provided an inappropriate and racist English translation for the Chinese term for “black foreigner” in its messaging app.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The AI system generated and exposed a user to a highly offensive racial slur (hate speech) due to biased associations in its translation algorithm.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The translation engine unfairly associated a neutral term for black individuals with a derogatory racial slur specifically in negative contexts, representing racial bias.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by an unexpected translation output from WeChat's neural machine translation system after it was deployed to users.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The translation software is a general-purpose utility tool integrated into a messaging app, posing low or negligible systemic risk to society under the EU AI Act framework.

AI system and alleged parties

• AI system: WeChat translation feature (Tencent)
• AI purpose: Translation
• Behaviour type: Tool
• Alleged developer: WeChat
• Alleged deployer: WeChat
• Alleged harmed parties: Black WeChat users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment where the translation engine produced a racial slur specifically for black individuals in negative contexts.

Directly caused: The AI system translated a neutral term for black foreigners into a derogatory slur when paired with negative adjectives, representing systemic racial bias in its outputs.

Inferred additional harm: The systemic association of black individuals with derogatory terms in negative contexts likely reinforced broader societal prejudices and discrimination against black people in China.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Inadequate Risk Assessment: Management failed to assess risks of offensive AI-generated content.
• Growth Prioritized Over Safety: Rapid scaling of translation service occurred without safety safeguards.

Technology

• Unconstrained Neural Translation: AI learned and literalized negative associations without output constraints.
• Missing Output Filter: System lacked a filter to block translation of known offensive slurs.

Data Inputs

• Biased Training Data: AI was trained on text containing racial slurs and negative associations.
• Negative Context Association: Training data linked neutral terms with slurs in negative contexts.

Human Factors

• No Human Oversight: Complete automation removed human review from the translation process.
• Lack of Cross-Cultural Awareness: Developers lacked sensitivity to global racial slurs and dynamics.

Process and Methods

• Inadequate Pre-release Testing: Testing failed to evaluate translation outputs under negative contexts.
• Reactive Moderation Process: Relied on user complaints rather than proactive safety checks.

Regulatory Environment

• Asymmetric Censorship Rules: Regulations mandated political censorship but neglected racial slurs.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from multiple reputable sources are highly consistent, detailing the exact phrase, the context of the translation error, the user's reaction, and the developer's response and subsequent fix.
• Ambiguities identified: None of significance.
• Alternative interpretations: None.

An unintentional translation error in WeChat's neural machine translation system translated a neutral Chinese phrase into a racial slur in negative contexts. While causing distress and highlighting algorithmic bias, the incident was quickly patched by Tencent, posing negligible threats to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The translation error was quickly patched by the developer, presenting no ongoing or immediate national security threat.
• Autonomy: Full autonomy. The neural machine translation engine generated and displayed translations directly to users without real-time human review.
• Novelty: Established threat. Algorithmic bias and offensive outputs from machine learning models trained on uncurated datasets are well-documented, established issues.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure impacts were reported in connection with this translation software incident.
• Information security: Negligible. The incident was an unintentional translation error rather than a coordinated information warfare campaign, state-sponsored disinformation, or intelligence compromise.
• Sovereignty: Negligible. There was no disruption to state authority, government decision-making, electoral systems, or constitutional processes.
• Economic security: Negligible. The incident did not involve theft of strategic technology, financial market manipulation, or disruption to critical supply chains.
• Societal stability: Minor. The translation engine generated a derogatory racial slur on a massive public platform, exposing systemic algorithmic bias and causing psychological distress, though manageable through standard software updates.