High-Toxicity Assessed on Text Involving Women and Minority Groups

Google's Perspective API, which assigns a toxicity score to online text, seems to award higher toxicity scores to content involving non-white, male, Christian, heterosexual phrases.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The Perspective API exhibits significantly higher false positive rates for toxicity when processing text containing minority identity descriptors compared to majority ones, representing unequal performance across demographic groups.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The system is highly vulnerable to simple adversarial perturbations, such as minor misspellings or punctuation changes, which drastically reduce its detection accuracy.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The issues of demographic bias and adversarial vulnerability arose unintentionally from the model's training data and architecture, manifesting after its deployment across various online platforms.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The Perspective API is a content moderation tool similar to a spam filter, which is classified under the EU AI Act as posing minimal or no risk to users and society.

AI system and alleged parties

• AI system: Perspective API (Google)
• AI purpose: Content Moderation; Sentiment Analysis
• Behaviour type: Tool
• Alleged developer: Google
• Alleged deployer: Google
• Alleged harmed parties: Women, Minority Groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Minor
• Differential treatment: direct Minor, indirect Substantial
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports describe how toxic content can easily bypass the filter using simple misspellings, and how actual toxic statements are rated as safe.

Indirectly caused: The API's failure to robustly detect toxic content allowed abusive comments (e.g., using bypassed words like 'idiiot') to remain visible on platforms using the tool.

Inferred additional harm: It is highly likely that thousands of toxic comments bypassed moderation filters across platforms utilizing the Perspective API due to these simple adversarial vulnerabilities.

Differential treatment

Reported: Yes, the reports explicitly describe how the API disproportionately flags identity-related terms associated with minority groups as toxic.

Directly caused: The API directly scored neutral self-identifications of minority groups (e.g., 'I am a gay black woman') as highly toxic (87%), while scoring majority identifiers as safe.

Indirectly caused: Online platforms using the API automatically flagged, hid, or suppressed comments from minority individuals, leading to unequal treatment of these groups in public forums.

Inferred additional harm: Widespread automated suppression of minority, LGBTQ+, and disabled voices across the internet on platforms relying on the Perspective API.

Civil rights

Reported: Yes, the reports describe how the algorithm's bias leads to the silencing and censorship of entire populations, impacting their freedom of expression.

Indirectly caused: Automated moderation systems using the API suspended or silenced users (such as a psychologist discussing racism or an African-American blogger) for using identity-related terms.

Inferred additional harm: Systemic, large-scale restriction of free speech and expressive rights for marginalized communities online due to biased automated moderation.

Epistemic

Reported: Yes, the reports describe how the tool generates highly inaccurate and misleading toxicity scores that misrepresent the nature of online comments.

Directly caused: The API produced false and misleading toxicity ratings, such as rating 'genocide is good' as safe (4% toxic) and neutral statements as highly toxic.

Indirectly caused: Publishers and researchers (such as Wired in its 'troll map' study) relied on these flawed scores, leading to incorrect public conclusions about online discourse.

Inferred additional harm: Widespread distortion of research and public understanding of online incivility due to reliance on a fundamentally flawed metric.

People affected

• Occurrences reported: 1
• People reportedly harmed: 4
• People reportedly exposed: 2000000

Potential causes

Management

• Premature Commercial Promotion: Promoted API to publishers without warning them of documented biases.
• Prioritizing Easy Solutions: Choosing easy NLP methods over robust systems that handle complex nuances.
• Inadequate Risk Assessment: Failed to assess the risk of silencing marginalized voices via biased scoring.

Technology

• Adversarial Vulnerability: Simple misspellings and punctuation bypass toxicity filters easily.
• Lack of Contextual Understanding: Fails to detect sarcasm, snark, and distinguishes quotes of hate speech poorly.
• Spurious Word Correlations: Flags benign words like 'die' or 'kill' regardless of actual toxic intent.

Data Inputs

• Biased Training Datasets: Models trained on comments from NYT and Wikipedia inherited systemic biases.
• Identity Descriptor Confounding: Identity terms like 'black' or 'gay' are heavily associated with toxicity.
• Out-of-Vocabulary Handling Flaws: Non-white names categorized as out-of-vocabulary, raising toxicity scores.

Human Factors

• Crowd-Worker Labeling Biases: Crowd-workers rating comments introduced subjective and cultural biases.
• Overreliance on Automated Moderation: Publishers deploy the API off-the-shelf without sufficient manual review.
• Culture of Civility Prioritization: Focusing on politeness over actual harm lets polite hate speech pass through.

Process and Methods

• Lack of Pre-processing Filters: Absence of spellcheckers allows simple adversarial workarounds to succeed.
• Inadequate Debiasing Techniques: Debiasing efforts failed to remove underlying representation biases.
• Flawed Performance Metrics: Evaluating model on benign settings ignored real-world adversarial behavior.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed, and empirical evidence from multiple independent sources (journalists, academic researchers, and security experts) demonstrating the API's demographic biases and adversarial vulnerabilities. The technical mechanisms of the failures (spurious correlations with identity terms, simple spelling bypasses) are clearly documented and verified by Jigsaw's own acknowledgments.
• Ambiguities identified: There is some ambiguity regarding the exact internal model architecture changes made over time, as the reports span from 2017 to 2020, and Jigsaw made minor updates to lower scores or mitigate biases.
• Alternative interpretations: One could argue the high toxicity scores for identity terms are purely statistical reflections of the toxic training data rather than 'built-in bias', though the resulting real-world harm of unequal performance remains the same.

Google's Perspective API exhibited significant algorithmic bias against minority groups and vulnerability to simple adversarial bypasses. While deployed at scale across major publishers, its impact is primarily limited to minor societal and civil rights concerns regarding online content moderation, posing negligible direct threats to national security.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: United States, Spain
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The algorithmic bias and adversarial vulnerability represent an ongoing, systemic technology issue rather than an immediate national security crisis.
• Autonomy: Human-supervised. The API automatically scores text toxicity, which is then integrated into platform moderation workflows that may include human oversight or automated thresholds.
• Novelty: Evolved capability. Represents a significant real-world scale deployment of NLP models, highlighting known algorithmic bias and robustness vulnerabilities on a massive scale.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were impacted or threatened by the content moderation API.
• Information security: Negligible. The incident involves a commercial content moderation tool and does not represent an active foreign information operation or intelligence compromise.
• Sovereignty: Negligible. No core government operations, electoral systems, or state authority functions were disrupted or compromised.
• Economic security: Negligible. While demonstrating vulnerabilities in Google's API, the incident did not result in strategic technology theft, financial system attacks, or critical economic damage.
• Societal stability: Minor. Algorithmic bias led to the disproportionate flagging and automated suppression of minority voices online, representing a minor impact on digital civil liberties.