AI Beauty Judge Did Not Like Dark Skin

In 2016, after artificial inntelligence software Beauty.AI judged an international beauty contest and declared a majority of winners to be white, researchers found that Beauty.AI was racially biased in determining beauty.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The AI system performed with lower accuracy and effectiveness for dark-skinned contestants, frequently discarding their photos due to lighting or failing to recognize their features due to underrepresentation in the training data.

Additional risk subdomains

• 1.1 Unfair discrimination and misrepresentation: The final selection of winners heavily favored light-skinned individuals, reinforcing biased and discriminatory standards of beauty.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The biased selection of beauty contest winners was an automated decision made by the deployed AI system, which was an unintended outcome of its training.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The AI system was used for an online beauty contest, which is an entertainment application posing low or negligible risks to society under the EU AI Act framework.

AI system and alleged parties

• AI system: Beauty.AI (Youth Laboratories)
• AI purpose: Image Scoring; Image Classification
• Behaviour type: Multi-agent
• Alleged developer: Youth Laboratories
• Alleged deployer: Youth Laboratories
• Alleged harmed parties: People with Dark Skin

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of individuals based on skin color and race caused directly by the incident.

Directly caused: The AI beauty contest algorithms selected 44 winners, of whom nearly all were white, only a handful were Asian, and only one had dark skin, despite many people of color submitting photos.

Indirectly caused: The biased results reinforced societal prejudices and exclusionary standards of beauty, sparking widespread controversy and public debate.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1500
• People reportedly exposed: 6000

Potential causes

Management

• Inadequate public relations outreach: Poor outreach in non-European countries led to a skewed entrant demographic.
• Lack of pre-launch bias testing: Management failed to test the system for racial bias prior to contest launch.

Technology

• Deep learning model limitations: Models reproduced human biases present in shared databases.
• Facial recognition failure in dim light: Algorithms discarded selfies of dark-skinned people if lighting was too dim.
• Biased reference database: MADIS compared selfies to databases of models and actors lacking diversity.

Data Inputs

• Underrepresentation of minorities: Training datasets lacked an adequate number of samples of people of color.
• Skewed contestant submission data: 75 percent of contest entrants were European and white, distorting the pool.
• Inadequate image quality control: System excluded dark-skinned images due to background and face color contrast.

Human Factors

• Entrenched human developer bias: Humans creating algorithms have entrenched biases that reproduce in the AI.
• Assumption of algorithmic neutrality: Belief that machines are objective counterparts led to lack of bias checks.

Process and Methods

• Shared biased training frameworks: Researchers shared training databases and frameworks without modifying them.
• Lack of proactive bias adjustments: No proactive adjustments were made to the algorithms to curb known biases.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent details across multiple sources regarding the Beauty.AI contest, the number of contestants (approx. 6,000), the number of winners (44), the specific algorithms used, and the technical reasons for the bias (lack of diverse training data and lighting issues). There are minor discrepancies in total contestant numbers in some reports (e.g., one mentioning 60,000 or 600,000), but the core facts of the incident are clear and undisputed.
• Ambiguities identified: There is a slight discrepancy in the reported number of contestants, with most reports stating 6,000, while one mentions 60,000 and another mentions 600,000.

The Beauty.AI incident in 2016 involved an online beauty contest where AI algorithms produced racially biased results due to unrepresentative training data. While it highlighted significant ethical concerns regarding algorithmic bias and discrimination, the incident was entirely within the entertainment domain and carried no national security implications.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Multiple nations
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident was a post-deployment commercial failure with no active or imminent security threat.
• Autonomy: Human-supervised. The AI algorithms evaluated images and selected winners autonomously, but under the overall supervision and hosting of the developers.
• Novelty: Established threat. Algorithmic bias due to unrepresentative training data is a well-known technical issue rather than a novel threat vector.

Impact by dimension

• Physical security: Negligible. No threats to physical systems, infrastructure, or human safety occurred as this was an online beauty contest.
• Information security: Negligible. The incident involved an entertainment app, with no intelligence compromise, classified data theft, or state-sponsored information warfare.
• Sovereignty: Negligible. No impact on state authority, government decision-making, or electoral processes.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or critical supply chain disruptions occurred.
• Societal stability: Negligible. While the incident demonstrated algorithmic bias and sparked public debate on discrimination, it posed no threat to societal stability, civil liberties, or national security.