Biased Sentiment Analysis

Google Cloud's Natural Language API provided racist, homophobic, amd antisemitic sentiment analyses.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The sentiment analyzer's accuracy and effectiveness differed significantly based on the demographic group mentioned, incorrectly labeling neutral minority identifiers as negative.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The system's outputs unfairly represented religious and sexual minorities by associating their identities with negative sentiment scores.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The bias was manifested through the outputs of the deployed AI system, which was trained on data that unintentionally introduced discriminatory associations.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The Cloud Natural Language API is a general-purpose natural language processing utility used for sentiment analysis, which poses low or negligible risk on its own, similar to spam filters or simple text processing applications.

AI system and alleged parties

• AI system: Cloud Natural Language API (Google)
• AI purpose: Sentiment Analysis; Content Moderation
• Behaviour type: Tool
• Alleged developer: Google
• Alleged deployer: Google
• Alleged harmed parties: Women, Minority Groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Substantial
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the reports describe how Google's Cloud Natural Language API produced biased sentiment scores, treating phrases identifying religious, ethnic, and sexual minorities differently and more negatively than majority groups.

Directly caused: The API directly assigned negative sentiment scores (ranging from -0.1 to -0.4) to neutral statements such as 'I'm a Jew', 'I'm homosexual', and 'I'm a gay black woman', while assigning positive or neutral scores to 'I'm Christian' or 'I'm straight'.

Indirectly caused: Because this API is used by third-party developers to analyze customer conversations, social media, and other texts, its biased outputs could indirectly lead to downstream discrimination or unfair filtering of minority users' content.

Inferred additional harm: It is likely that developers using this API in production environments unknowingly subjected their users to biased content moderation or sentiment filtering, potentially silencing or marginalizing minority voices at scale.

People affected

• Occurrences reported: 1

Potential causes

Management

• Code of Conduct Mismatch: Company non-discrimination policies were not integrated into AI development.
• Rapid Public Beta Release: Prioritized launching the public beta before ensuring model inclusivity.

Technology

• Flawed Sentiment Model: The model mapped demographic identifiers to negative sentiment scores.
• Lack of De-biasing Tech: No reliable technical methods existed to completely remove bias from NLP models.

Data Inputs

• Biased Training Data: Model trained on societal texts containing historical prejudices and biases.
• Polluted External Corpora: Web-scraped data contained negative associations for minority terms.
• Word Connotation Imbalance: Terms like 'Jew' appeared more in negative contexts than 'Jewish'.

Human Factors

• Societal Prejudices: Human biases in source texts were absorbed and mirrored by the AI system.
• User Bias Amplification: Online content creators generated biased text that polluted training datasets.

Process and Methods

• Inadequate Data Vetting: Lack of thorough vetting of the training corpus before API deployment.
• Insufficient Pre-release Testing: The public beta was launched without detecting obvious demographic biases.

Regulatory Environment

• Lack of AI Bias Standards: No regulatory frameworks or standards existed for auditing algorithmic bias.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are consistent, from multiple reputable tech journalism sources, and include direct confirmation and apologies from Google. The nature of the bias is clearly demonstrated with reproducible inputs and outputs.
• Ambiguities identified: The exact training dataset (corpus) used by Google is not disclosed, though it is widely attributed to general web crawls and books.
• Alternative interpretations: None. The bias is clear and acknowledged by the developer.

Google's Cloud Natural Language API was found to produce biased sentiment scores, negatively labeling religious, ethnic, and sexual minority identifiers. While the incident highlights critical issues regarding algorithmic bias and potential downstream discrimination, it poses negligible direct threat to national security, infrastructure, or sovereignty.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing strategic concern regarding algorithmic bias rather than an active, time-sensitive national security crisis.
• Autonomy: Human-controlled. The AI operates strictly as a tool, processing text inputs and returning scores without autonomous decision-making or execution capabilities.
• Novelty: Established threat. Algorithmic bias and representative skew in machine learning training datasets are well-documented and established issues in AI development.

Impact by dimension

• Physical security: Negligible. No physical systems, infrastructure, or human safety were impacted or threatened by this software bias.
• Information security: Negligible. No evidence of coordinated disinformation campaigns, intelligence compromise, or state-sponsored information operations.
• Sovereignty: Negligible. State authority, territorial control, and core government decision-making processes were entirely unaffected.
• Economic security: Negligible. The incident involved a public API defect with no strategic technology theft, financial market disruption, or loss of national competitive advantage.
• Societal stability: Minor. The API exhibited systematic bias by assigning negative sentiment scores to minority identity terms, presenting a minor risk of automated discrimination if deployed downstream.