FaceApp Racial Filters

FaceApp is criticized for offering racist filters.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The AI system misrepresented users of color by automatically whitening their skin to make them look 'hot' and generated offensive, stereotyped representations of different races through its ethnicity filters.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The neural network exhibited a lack of robustness due to training set bias, failing to perform equitably across different demographic groups.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was primarily driven by the biased outputs of FaceApp's neural network and the automated generation of racially stereotyped filters.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The system is an entertainment application used for simple photo filtering, which poses low or negligible risk to society under the EU AI Act guidelines.

AI system and alleged parties

• AI system: FaceApp (Wireless Lab)
• AI purpose: Image Retouching; Deepfake Image Generation
• Behaviour type: Tool
• Alleged developer: FaceApp
• Alleged deployer: FaceApp
• Alleged harmed parties: Minority Groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The reports explicitly describe the creation and spread of racially offensive content generated by the app's filters.

Directly caused: The AI generated photorealistic images that altered users' skin tones and facial features to fit racial stereotypes, which was widely compared to digital blackface and yellowface.

Indirectly caused: These offensive generated images were shared widely across social media platforms like Twitter and Instagram, exposing a broader audience to racially insensitive content.

Inferred additional harm: Millions of social media users were likely exposed to these offensive generated images during the viral cycles of the app.

Differential treatment

Reported: The reports explicitly describe differential treatment of individuals based on their race and skin tone.

Directly caused: The 'hot' filter systematically lightened the skin of dark-skinned users, implying that Eurocentric features and lighter skin are the standard of beauty.

Inferred additional harm: The widespread use of the app reinforced colorism and racial bias among its millions of users globally.

Psychological

Reported: The reports explicitly describe psychological distress, offense, and anger experienced by users due to the racially insensitive filters.

Directly caused: Users, including a 16-year-old girl, reported being 'infuriated', 'shocked', and offended by filters that altered their racial appearance or whitened their skin.

Inferred additional harm: It is highly likely that thousands of other minority users experienced feelings of exclusion, offense, and psychological distress when using the app's biased filters.

People affected

• Occurrences reported: 2
• People reportedly harmed: 15
• People reportedly exposed: 40000000

Potential causes

Management

• Prioritizing Novelty Over Safety: Management focused on viral growth and photorealism over ethics.
• Defensive Initial Response: CEO initially defended the filters as equal instead of removing them.

Technology

• Biased Neural Network: The deep learning model associated whiter skin with hotness.
• Stereotyping Algorithms: Filters altered features like nose and lips based on racial stereotypes.

Data Inputs

• Homogeneous Training Data: Training set was biased, containing mostly photos of white people.
• Lack of Diverse Datasets: The AI lacked representative images for non-white ethnicities.

Human Factors

• Cultural Insensitivity: Developers lacked awareness of blackface and racial parody history.
• Algorithmic Overreliance: Assuming neural net outputs are objective and equal in all aspects.

Process and Methods

• Inadequate Pre-release Testing: The app was launched without testing for racial bias or offense.
• Poor Risk Assessment: No process to evaluate the social impact of ethnicity-altering AI.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly consistent across multiple reputable news outlets, providing clear details on the app's functionality, the nature of the public backlash, and statements from the company's CEO.
• Ambiguities identified: The exact demographic breakdown of the training dataset is not specified beyond the CEO's admission of 'training set bias'.

FaceApp's release of racially biased and offensive filters sparked global public backlash, highlighting issues of algorithmic discrimination and consumer data privacy. While the incident caused notable societal offense and exposed millions of users to biased AI outputs, its direct threat to national security remains negligible to minor.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global users
• Alleged perpetrator: Wireless Lab

Threat characteristics

• Imminence: Long-term. The incident represents a long-term societal concern regarding algorithmic bias and data privacy rather than an immediate national security crisis.
• Autonomy: Human-controlled. The AI system functions as a passive tool, executing specific photo-editing tasks only when prompted and configured by human users.
• Novelty: Evolved capability. Represented a significant scaling of deep generative neural networks applied directly to consumer products, exposing millions to algorithmic bias.

Impact by dimension

• Physical security: Negligible. No physical systems, infrastructure, or human safety elements were threatened or affected by this photo-editing application.
• Information security: Negligible. Although privacy experts raised concerns about the Russian developer's data retention, there was no reported intelligence compromise or information warfare operation.
• Sovereignty: Negligible. The incident did not impact state authority, electoral systems, or core government decision-making processes.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or critical supply chain disruptions occurred as a result of this consumer app update.
• Societal stability: Minor. The release of racially biased and offensive filters caused widespread public outcry and highlighted algorithmic discrimination at scale, though it did not threaten national stability.