AI Spam Filters Allegedly Block Legitimate Emails Based on Biased Keyword Detection

AlgorithmWatch tested spam filtering algorithms across Gmail, Yahoo, Outlook, GMX, and LaPoste. Their findings reportedly showed that Microsoft Outlook’s spam filter flagged emails based on specific keywords that led to racial and content-based biases blocking legitimate communications. Emails mentioning Nigeria or containing certain financial and sexual health terms were found to be disproportionately marked as spam.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The spam filtering algorithms perform with lower accuracy for emails associated with specific demographic groups (such as Nigerians), leading to legitimate communications being disproportionately blocked.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The spam filters lack robustness, failing to distinguish benign context from spam indicators for common words like 'sex' or 'loan'.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The discriminatory filtering was caused by deployed machine learning algorithms and automated rules unintentionally misclassifying legitimate emails based on biased training data.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk. The EU AI Act explicitly categorizes 'simple applications like spam filters' as posing minimal or no risk, exempting them from additional regulatory obligations.

AI system and alleged parties

• AI system: Microsoft Outlook spam filter, SpamAssassin (Apache Software Foundation, Microsoft)
• AI purpose: Spam Filtering; Content Moderation
• Behaviour type: Tool
• Alleged developer: Yahoo, Microsoft, LaPoste, Google, GMX
• Alleged deployer: Yahoo, Outlook, LaPoste, GMX, Gmail
• Alleged harmed parties: Yahoo! Mail users, Microsoft Outlook users, LaPoste users, GMX users, Gmail Users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment where emails containing words like 'Nigeria' or 'sex' were disproportionately marked as spam compared to identical emails without those words.

Directly caused: Legitimate emails, such as an internship application from a Nigerian student, were blocked and sent to spam solely due to the inclusion of the word 'Nigeria'.

Inferred additional harm: It is highly likely that thousands of individuals from Nigeria or those discussing sensitive but legitimate topics (like sex education) routinely face systemic communication barriers due to these biased filters.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 10

Potential causes

Management

• Lack of Corporate Transparency: Microsoft does not share training data or comment on filter bias.
• Ineffective Community Governance: SpamAssassin delegates rule issues to an unvetted community.

Technology

• Biased ML Classification: ML algorithms over-rely on single words like 'Nigeria' to classify spam.
• Flawed Default Configurations: Default rules in filters like SpamAssassin block specific country names.

Data Inputs

• Imbalanced Training Data: Public corpora contain Nigerian emails only in spam folders, not in ham.
• Unvetted User Submissions: User-submitted training data is not checked for unfair discrimination.

Human Factors

• Lack of Developer Diversity: Project committees lack cultural diversity, leading to bias blind spots.
• Individual Biases: Maintainers with biased views influence community rules and standards.

Process and Methods

• Lack of Algorithmic Auditing: Spam filters are largely unaudited for discriminatory biases.
• Outdated Training Corpora: Using 15-year-old training datasets that preserve historical biases.

Regulatory Environment

• Absence of Filtering Standards: No regulatory frameworks exist to govern bias in spam filtering.
• Lack of Independent Oversight: No external bodies audit private email providers for discrimination.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, empirical evidence from a structured experiment by AlgorithmWatch demonstrating the biased behavior of Outlook and SpamAssassin. The mechanics of the failure (keyword-based filtering driven by machine learning on biased datasets) are well-explained and uncontroversial.
• Ambiguities identified: The exact machine learning models and training datasets used by Microsoft remain proprietary and undisclosed.
• Alternative interpretations: None. The empirical results clearly demonstrate the keyword-triggered spam classification.

An investigation revealed that Microsoft Outlook and SpamAssassin spam filters exhibit racial and content-based biases, disproportionately flagging emails containing keywords like 'Nigeria' or financial terms. While representing a systemic bias issue with minor human rights implications, the incident does not pose a direct threat to national security.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: Nigeria
• Other affected: Global users of Microsoft Outlook and SpamAssassin
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing, systemic software bias issue rather than an immediate national security crisis.
• Autonomy: Full autonomy. The spam filtering algorithms autonomously decide whether to deliver or block emails without real-time human intervention.
• Novelty: Established threat. Algorithmic bias and training data limitations in machine learning models are well-documented and established issues.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety concerns were impacted by the spam filtering algorithms.
• Information security: Negligible. No evidence of foreign intelligence compromise or deliberate information warfare operations targeting national security.
• Sovereignty: Negligible. While some political communications like speeches were filtered, there was no systematic threat to state authority or core government operations.
• Economic security: Negligible. No strategic threat to national economic stability, critical supply chains, or technological competitive advantage was identified.
• Societal stability: Minor. Systematic bias in spam filters represents a minor discriminatory impact against specific nationalities and topics, but does not threaten overall societal stability.