Genderify’s AI to Predict a Person’s Gender Revealed by Free API Users to Exhibit Bias

A company's AI predicting a person's gender based on their name, email address, or username was reported by its users to show biased and inaccurate results.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The Genderify tool automated sexist stereotypes and misrepresented individuals by associating professional titles with men and erasing non-binary identities.

Additional risk subdomains

• 1.3 Unequal performance across groups: The system performed poorly for female names and usernames, and completely failed to accommodate non-binary individuals.
• 7.3 Lack of capability or robustness: The tool exhibited bizarre and highly inaccurate predictions, such as classifying the word 'woman' as 96% male.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by biased predictions generated by the Genderify AI system, which was deployed publicly before its severe stereotyping and inaccuracies were realized.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a profiling tool that poses moderate risks of bias and discrimination, requiring transparency, though it was intended for marketing database segmentation.

AI system and alleged parties

• AI system: Genderify
• AI purpose: Gender Identification; Marketing Analytics Generation
• Behaviour type: Tool
• Alleged developer: Genderify
• Alleged deployer: Genderify
• Alleged harmed parties: Genderify customers, gender minority groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report describes toxic outputs in the form of highly biased and sexist gender predictions generated by the AI.

Directly caused: The AI directly generated biased and sexist outputs, such as predicting 'stupid' as 61.7% female and 'professor' as 98.4% male.

Differential treatment

Reported: The report explicitly describes differential treatment and the potential for discriminatory outcomes in marketing and employment.

Directly caused: The tool directly discriminated against women and non-binary individuals by reinforcing sexist stereotypes and forcing a binary classification.

Indirectly caused: The report notes that if used by businesses, the tool would have resulted in discriminatory targeted advertising and the exclusion of individuals from employment opportunities.

Inferred additional harm: Any businesses that integrated the API prior to its shutdown would have systematically misclassified and discriminated against female and non-binary customers.

People affected

• Occurrences reported: 1
• People reportedly harmed: 5
• People reportedly exposed: 1000

Potential causes

Management

• Premature Public Release: Launching on Product Hunt before validating safety and accuracy.
• Prioritizing Launch over Safety: Focusing on market novelty instead of addressing fundamental bias risks.
• Defensive Risk Management: Initial response minimized harm by pointing to other biased market solutions.

Technology

• Biased Algorithmic Predictions: Algorithm predicted occupations like doctor and professor as heavily male.
• Inflexible Binary Classification: The model forced users into binary categories, ignoring non-binary identities.
• Erroneous Association Logic: Adding titles like Dr. or words like woman skewed predictions to male.

Data Inputs

• Biased Training Data: AI was trained on historical data reflecting societal gender biases.
• Inadequate Input Proxies: Using names, usernames, or emails as reliable proxies for gender identity.
• Reliance on Binary Databases: Model relied on existing binary name and gender databases.

Human Factors

• Developer Overconfidence: Belief that a self-learning AI would inherently resolve existing biases.
• User Backlash and Outcry: Public outrage on Twitter regarding offensive and sexist predictions.
• Overreliance on User Feedback: Relying on LGBTQ+ community feedback post-launch to fix core design flaws.

Process and Methods

• Lack of Pre-launch Bias Testing: The tool was released publicly without evaluating sexist or absurd outputs.
• Flawed Product Design: Automating gender assumptions at scale without consent or validation.
• Inadequate Feedback Mechanisms: No robust pre-release mechanism to capture and address algorithmic errors.

Regulatory Environment

• Lack of Industry Standards: No standard regulations or mandates preventing biased AI deployments.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly consistent, detailing the tool's launch, specific biased outputs, public reaction, and subsequent shutdown. The facts are clear and uncontested.

Genderify, an AI tool designed to predict gender from names and emails, was shut down hours after launch due to public backlash over severe gender and professional biases. The incident represents a commercial product failure with negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The service was permanently shut down immediately after launch, presenting no ongoing or near-term national security threat.
• Autonomy: Human-controlled. The AI tool operated as a static API predicting gender from user-provided inputs, requiring human integration and execution.
• Novelty: Established threat. Algorithmic bias and demographic misclassification in machine learning models are well-documented and established issues.

Impact by dimension

• Physical security: Negligible. The incident involved a software utility for gender prediction and had no connection to physical systems, infrastructure, or kinetic threats.
• Information security: Negligible. No intelligence capabilities, classified models, or systematic information operations were compromised or targeted in this incident.
• Sovereignty: Negligible. The tool was a commercial marketing API and did not impact state authority, electoral processes, or core government decision-making.
• Economic security: Negligible. While the business shut down, the incident represents a minor commercial failure with no strategic economic or technological security implications for any nation.
• Societal stability: Negligible. The tool exhibited sexist biases and erased non-binary identities, but its immediate shutdown within hours of launch prevented any widespread societal-scale harm or systemic discrimination.