GPT-4o's Chinese Tokens Reportedly Compromised by Spam and Pornography Due to Inadequate Filtering

OpenAI's GPT-4o was found to have its Chinese token training data compromised by spam and pornographic phrases due to inadequate data cleaning. Tianle Cai, a Ph.D. student at Princeton University, identified that most of the longest Chinese tokens were irrelevant and inappropriate, primarily originating from spam and pornography websites. The polluted tokens could lead to hallucinations, poor performance, and potential misuse, undermining the chatbot's reliability and safety measures.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The tokenizer fails to perform reliably under non-English inputs due to a mismatch between tokenizer and model training data, exposing the system to errors and jailbreak vulnerabilities.

Additional risk subdomains

• 1.3 Unequal performance across groups: The tokenizer's pollution specifically degrades performance and security for Chinese-speaking users compared to English speakers.
• 2.2 AI system security vulnerabilities and attacks: The polluted tokens represent a security vulnerability that malicious actors can exploit to bypass safety guardrails.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk of model exploitation and hallucination is caused by the deployed GPT-4o tokenizer behaving in an unexpected and vulnerable manner.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a general-purpose chatbot (GPT-4o), which falls under Risk Level 3 due to transparency obligations for AI systems interacting with humans.

AI system and alleged parties

• AI system: GPT-4o (OpenAI)
• AI purpose: Chatbot; Translation
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: OpenAI, GPT-4o
• Alleged harmed parties: Researchers, OpenAI users, OpenAI, Chinese-speaking users of ChatGPT

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly exposed: 5

Potential causes

Management

• Prioritizing Cost Over Safety: Focused on token compression and cost reduction over data safety reviews.
• Insufficient Pre-Release Testing: Failed to review the public token library for non-English spam before release.

Technology

• Tokenizer-LLM Dataset Mismatch: Tokenizer trained on uncleaned data while LLM was trained on filtered data.
• Undefined Model Behavior: Under-trained tokens trigger random signals and hallucinations in the model.

Data Inputs

• Unfiltered Chinese Training Corpus: Training data contained high volumes of spam, porn, and gambling phrases.
• Search-Engine Spam Hijacking: Spam bots hijacked legitimate web content, which was crawled into datasets.

Process and Methods

• Lack of Non-English Data Cleaning: No proper data filtering was applied to Chinese training datasets.
• Early-Stage Tokenizer Training: Tokenizer training occurs early without the rigorous safety filters of LLMs.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, detailed technical explanations of the tokenizer issue, including specific examples of polluted tokens, the mechanism of the vulnerability, and successful replication of jailbreaks by named researchers.
• Ambiguities identified: It is slightly unclear how many other languages besides Chinese and Korean are affected, and whether the multi-modal aspects of GPT-4o share the same vulnerability.
• Alternative interpretations: The incident could be interpreted primarily as a data quality/cleaning failure (Human) rather than an AI robustness failure, though the resulting risk manifests as an AI system vulnerability.

A data-cleaning failure in OpenAI's GPT-4o tokenizer resulted in Chinese and Korean token libraries being polluted with spam and pornographic terms. This vulnerability allows users to bypass safety guardrails and causes model hallucinations. While representing a notable security gap in a major AI system, its immediate national security impact remains minor and manageable under standard procedures.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: China, South Korea
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Near-term. The vulnerability is active in a widely deployed commercial model, presenting a near-term risk of exploitation until fully patched.
• Autonomy: Human-controlled. The AI system functions as an assistant, and exploiting the vulnerability requires active human prompting and control.
• Novelty: Evolved capability. While LLM jailbreaking is an established threat, utilizing data-polluted tokenizers to bypass safety guardrails represents an evolved vulnerability vector.

Impact by dimension

• Physical security: Minor. The tokenizer vulnerability allowed researchers to bypass safety guardrails to request dangerous instructions such as bomb-making, though secondary safety filters blocked the final output.
• Information security: Minor. Model hallucinations and safety bypasses could be exploited for automated information manipulation, but no active foreign intelligence or disinformation campaign is reported.
• Sovereignty: Negligible. No direct threat to state authority, electoral systems, or core government decision-making processes was identified in the incident.
• Economic security: Minor. Reveals quality control and security vulnerabilities in a major commercial AI model, reflecting technological robustness challenges, but caused no direct economic disruption.
• Societal stability: Minor. The presence of spam, gambling, and pornographic tokens disproportionately degrades model performance and safety for Chinese and Korean language users, though without causing civil unrest.