FaceApp Predicted Different Genders for Similar User Photos with Slight Variations

FaceApp’s algorithm was reported by a user to have predicted different genders for two mostly identical facial photos with only a slight difference in eyebrow thickness.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The AI systems exhibited unequal performance and accuracy in gender classification that is dependent on demographic group presentation, leading to the alienation and distress of a gender-nonconforming user.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The user relied on consumer entertainment classification apps for personal gender validation, which triggered emotional distress when the systems failed to perform robustly.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The harm was caused by the classification decisions of the deployed AI systems, which produced unexpected negative psychological outcomes for the user.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The AI systems involved are consumer entertainment applications used for photo editing and simple age/gender estimation, which carry no additional regulatory obligations under the Act.

AI system and alleged parties

• AI system: FaceApp
• AI purpose: Gender Identification; Image Classification
• Behaviour type: Tool
• Alleged developer: FaceApp
• Alleged deployer: FaceApp
• Alleged harmed parties: FaceApp users, FaceApp transgender users, FaceApp non-binary presenting users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment where the user was classified differently based on minor physical variations.

Directly caused: The AI systems treated the user differently by classifying them as male or female based purely on minor features like eyebrow thickness or wearing glasses.

Inferred additional harm: Other users with non-stereotypical gender presentations are likely subjected to similar inconsistent and biased classifications by these systems.

Psychological

Reported: The report explicitly describes psychological distress, gender dysphoria, and feeling trapped as a result of the algorithmic classifications.

Directly caused: The AI's inconsistent and stereotypical gender classifications directly triggered the user's gender dysphoria and emotional distress.

Indirectly caused: The resulting distress and feeling of being trapped led the user to overpluck their eyebrows and stop wearing glasses to conform to the AI's patterns.

Inferred additional harm: It is highly likely that other transgender and gender-nonconforming users experience similar psychological distress and dysphoria when interacting with these non-robust classification systems.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Neglecting Mental Risks: Developers prioritize viral engagement over the mental health risks of users.

Technology

• Simplistic Heuristics: Model relies on superficial features like brow thickness to classify gender.
• Fragile Vision Models: Minor changes like wearing glasses drastically alter gender and age outputs.

Data Inputs

• Biased Training Data: Dataset correlates thick eyebrows with males and thin eyebrows with females.
• Poor Representation: Underrepresentation of non-binary or transgender features in training data.

Human Factors

• Validation Seeking: User relies on AI apps for gender affirmation, triggering dysphoria.
• Behavioral Modification: User alters physical appearance like overplucking brows to satisfy the AI.

Process and Methods

• Inadequate Bias Testing: Lack of robust testing for demographic variations and superficial features.
• No Safety Guardrails: Lack of warnings about model limitations on sensitive classifications.

Regulatory Environment

• Absence of Bias Standards: No regulatory standards governing gender classification or bias in consumer AI.

Information quality

• Classification confidence: High
• Reason for confidence: The report is a direct, first-hand account detailing the user's specific interactions with the AI systems and the resulting psychological impact. The behavior of the systems and the user's response are clearly described without conflicting details.
• Ambiguities identified: The exact technical specifications of the classification models used by FaceApp and the how-old site are not detailed in the report.
• Alternative interpretations: The incident could be interpreted purely as a user experience issue rather than an AI safety failure, but the psychological harm and behavioral modification point to a safety risk in demographic classification.

A consumer experienced gender dysphoria and altered their appearance due to inconsistent gender and age classifications by FaceApp and how-old. These classification errors in consumer entertainment apps present negligible national security implications across all assessed categories.

• Overall national security impact: Negligible
• Scope: Unknown
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents a localized consumer software issue rather than an impending national security threat.
• Autonomy: Human-controlled. The AI system operates as a tool, performing classification on images uploaded directly by the user.
• Novelty: Established threat. Inconsistent and biased algorithmic classifications in consumer-facing AI models are well-documented phenomena.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic targets were impacted by this consumer photo-editing application.
• Information security: Negligible. No intelligence operations, disinformation campaigns, or classified data compromises were associated with this incident.
• Sovereignty: Negligible. The incident involves a consumer app and does not impact state authority, electoral systems, or government functions.
• Economic security: Negligible. There are no indications of threats to financial systems, strategic technologies, or economic stability.
• Societal stability: Negligible. While the app caused individual psychological distress, it does not represent a large-scale threat to national social cohesion or population safety.