Twitter’s Image Cropping Tool Allegedly Showed Gender and Racial Bias

Twitter's photo cropping algorithm was revealed by researchers to favor white and women faces in photos containing multiple faces, prompting the company to stop its use on mobile platform.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The image-cropping algorithm performed with unequal accuracy and effectiveness across different demographic groups, systematically favoring lighter-skinned individuals over Black individuals.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The automated cropping caused representational harm by denying users the ability to represent themselves as they wished on the platform.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The bias was an unexpected and unintended consequence of the deployed saliency algorithm's design and training data.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The image-cropping algorithm is used for media presentation on a social media platform, which falls under simple applications with low risk to safety, though it does raise ethical and bias concerns.

AI system and alleged parties

• AI system: Twitter image cropping algorithm (Twitter)
• AI purpose: Image Cropping; Image Scoring
• Behaviour type: Tool
• Alleged developer: Twitter
• Alleged deployer: Twitter
• Alleged harmed parties: Twitter Users, Twitter non-white users, Twitter non-male users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Negligible
• Civil rights: direct Substantial, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report mentions Twitter offering cash prizes of up to $3,500 for its bias bounty competition.

Directly caused: Twitter paid out bounty prizes (up to $3,500 per winning entry) to researchers who identified algorithmic biases.

Inferred additional harm: Twitter likely incurred internal operational and engineering costs to audit, modify, and eventually retire the algorithm.

Differential treatment

Reported: Yes, the reports explicitly describe differential treatment of individuals based on race, gender, age, disability, and language script.

Directly caused: The algorithm favored white individuals over Black individuals (e.g., displaying white women 64% of the time over Black men), favored younger/slimmer faces, and preferred Latin script over Arabic.

Inferred additional harm: Millions of minority users experienced systematic underrepresentation and differential visibility on the platform due to the automated cropping.

Civil rights

Reported: Yes, the reports describe concerns regarding representation harm and discrimination against marginalized groups.

Directly caused: The algorithm systematically marginalized Black users, the elderly, and people with disabilities by cropping them out of image previews.

Inferred additional harm: The systemic bias infringed upon the rights of equal treatment and non-discrimination for millions of users on a major public communications platform.

People affected

• Occurrences reported: 1
• People reportedly harmed: 100
• People reportedly exposed: 300000000

Potential causes

Management

• Undefined AI Authority: Unclear who had final authority to decide if AI is too unfair for use.
• Prioritizing Automation: Management prioritized automated speed over user choice and representation.

Technology

• Saliency-based Neural Network: The algorithm crops images based on predicted visual interest, creating bias.
• Contrast and Feature Bias: Algorithm is drawn to high contrast of light skin against dark backgrounds.

Data Inputs

• Biased Eye-Tracking Data: Model was trained on eye-tracking data that can encode human biases.
• Beauty Filter Skewed Inputs: Beauty filters skew saliency scores in favor of lighter, younger faces.
• Limited Evaluation Datasets: Evaluation datasets excluded nonbinary or mixed-race demographic groups.

Human Factors

• Potential Male Gaze Bias: Eye-tracking from straight male participants may encode sexualized views.
• Loss of User Agency: Automated cropping prevents users from choosing how to represent themselves.

Process and Methods

• Inadequate Initial Testing: Initial tests used too few images to detect demographic and gender bias.
• Lack of Ethical Review Standards: No standard fairness metrics were available during model experimentation.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed, including Twitter's own published scientific study, statistical results of demographic parity tests, and outcomes of a public bug bounty program. The technical mechanism (saliency tracking) and the nature of the bias are clearly documented.
• Ambiguities identified: Some initial user claims about the 'male gaze' (cropping women's chests) were not replicated in Twitter's systematic study, though users and bounty hunters still reported issues with beauty filters and other biases.
• Alternative interpretations: The bias could be interpreted purely as a data representation issue (contrast and lighting in training datasets) rather than systemic racial prejudice in the code itself.

Twitter's automated image-cropping algorithm exhibited systemic racial, gender, and demographic biases, favoring lighter-skinned individuals. While the incident affected millions of users globally and highlights significant challenges in AI fairness and representation, it has negligible direct national security implications and was resolved through corporate audits and system updates.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Multiple nations
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The issue was identified, audited, and mitigated by the platform, representing a long-term algorithmic bias concern rather than an active national security crisis.
• Autonomy: Full autonomy. The algorithm automatically determined saliency and executed image crops without human intervention before display.
• Novelty: Established threat. Algorithmic bias and demographic disparities in computer vision and facial recognition models are well-documented, established issues.

Impact by dimension

• Physical security: Negligible. No threats to physical systems, critical infrastructure, or human safety were reported or observed.
• Information security: Negligible. The incident involved an automated cropping tool and did not compromise intelligence capabilities or involve systematic disinformation campaigns.
• Sovereignty: Negligible. No impact on government functions, state authority, or electoral systems.
• Economic security: Negligible. No threats to strategic industries, financial systems, or national technological competitive advantage.
• Societal stability: Minor. The algorithm caused systemic representational bias affecting millions of users on a major platform, though it did not pose a threat to national security or cause civil unrest.