Networking Platform Giggle Employs AI to Determine Users’ Gender, Allegedly Excluding Transgender Women

A social networking platform, Giggle, allegedly collected, shared to third-parties, and used sensitive information and biometric data to verify whether a person is a woman via facial recognition, which critics claimed to be discriminatory against women of color and harmful towards trans women.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The facial recognition software exhibited unequal performance across demographic groups, specifically misclassifying darker-skinned women at a much higher rate of 22.5% and failing to verify transgender women.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The app's privacy policy allowed the collection and sharing of highly sensitive personal data including sexual practices and health information.
• 1.1 Unfair discrimination and misrepresentation: The platform's onboarding process was intentionally designed and used to systematically exclude transgender women.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident involves the post-deployment failure of the Kairos facial recognition AI to accurately classify users' gender, resulting in unintentional discriminatory exclusions.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system uses biometric identification and categorization of natural persons, which has significant implications for fundamental rights and public interests.

AI system and alleged parties

• AI system: Kairos facial-recognition AI (Kairos)
• AI purpose: Gender Identification; Face Recognition
• Behaviour type: Tool
• Alleged developer: Kairos
• Alleged deployer: Giggle
• Alleged harmed parties: women of color, trans women

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Minor
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and disparaging content targeting trans people on the platform and on social media.

Directly caused: Users on the Giggle app posted disparaging comments about trans people in the general discussion section.

Indirectly caused: The controversy led to toxic social media exchanges, including the founder sharing a tweet calling a trans user a 'beggar' and posting a collage to make fun of trans users.

Inferred additional harm: It is likely that additional transphobic and toxic content was generated within the app's unmonitored spaces before accounts were sorted.

Differential treatment

Reported: The report explicitly describes systematic differential treatment of transgender women and women of color.

Directly caused: The Kairos facial recognition software misgendered darker-skinned females at a rate of 22.5% and failed to verify transgender women, leading to their exclusion from the app.

Indirectly caused: The app's founder used the system's outputs and manual intervention to ban and remove transgender women from the platform.

Inferred additional harm: Numerous women of color and transgender women were likely denied access to the social platform due to the biased facial recognition algorithm.

Civil rights

Reported: The report does not explicitly describe legal violations of civil rights, but notes that advocates view the exclusion as a violation of trans people's rights and safety.

Indirectly caused: The exclusion of trans women from a public social platform based on gender identity represents a discriminatory practice that impacts civil rights to equal access.

Privacy

Reported: The report explicitly describes privacy concerns regarding the app's extensive data collection and sharing policies.

Directly caused: The app's privacy policy allowed the collection and third-party sharing of highly sensitive user data, including images, location, sexual practices, criminal records, and private health information.

Inferred additional harm: Users who signed up for the app may have had their sensitive personal and biometric data shared with third-party facial recognition providers and marketers without fully understanding the scope of the data collection.

Psychological

Reported: The report describes psychological harm, noting that trans advocates state the exclusionary ideology and misgendering create real-world harm and distress for trans people.

Directly caused: Transgender users experienced distress and feelings of exclusion when rejected by the app's gender-verification software or targeted with disparaging comments.

Indirectly caused: The public debate and online harassment surrounding the app's launch and the founder's comments caused distress to trans individuals.

Inferred additional harm: It is likely that many other trans women and women of color who were rejected by the app experienced feelings of exclusion, anxiety, and psychological distress that were not individually documented.

People affected

• Occurrences reported: 1
• People reportedly harmed: 5
• People reportedly exposed: 1000

Potential causes

Management

• Active Exclusion Policy: Management explicitly designed onboarding to gatekeep and exclude trans women.
• Dismissal of Algorithmic Bias: Management dismissed concerns about racism and bias in facial analysis.

Technology

• Biased Facial Recognition: Kairos AI misgendered darker-skinned females 22.5% of the time.
• Flawed Bone Structure Proxy: Using facial bone structure as a biological proxy for gender is inaccurate.
• Easily Bypassed Filter: Users could bypass rejection by simply submitting another selfie.

Data Inputs

• Eurocentric Training Datasets: AI training data favored lighter skin tones and Eurocentric facial features.
• Single Selfie Input Limitation: Verification relied on a single selfie, leading to high classification error.

Human Factors

• Transphobic Ideology of Founder: Founder intentionally chose to exclude trans women from the platform.
• User Frustration and Backlash: Users targeted the app with negative reviews due to unfair exclusion.

Process and Methods

• Inadequate Algorithmic Testing: The app launched despite known high error rates for diverse demographics.
• Arbitrary Manual Interventions: The CEO manually banned users based on social media callouts.
• Intrusive Data Collection: Privacy policy allowed sharing sensitive biometric data with third parties.

Regulatory Environment

• Weak App Store Policy Enforcement: App stores restored the app despite complaints of bias and exclusion.
• Lack of Biometric Regulation: Absence of laws restricting biometric gender categorization on social apps.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent details about the app's functionality, the specific AI vendor (Kairos), the nature of the bias (Eurocentric and trans-exclusionary), and the privacy policy concerns. The scientific limitations of the facial recognition software are backed by a cited MIT study.
• Ambiguities identified: The exact number of users affected or excluded is not specified.
• Alternative interpretations: None. The primary dispute is ideological (the founder's defense of the exclusion) rather than factual regarding how the AI functioned.

The Giggle social media app utilized biased facial recognition software for gender verification, resulting in discriminatory exclusion of trans women and women of color, coupled with highly invasive data collection practices. While raising significant ethical and privacy concerns, the incident remains a localized commercial issue with negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Giggle platform founder

Threat characteristics

• Imminence: Long-term. Represents ongoing strategic and regulatory concerns regarding AI bias rather than an active national security crisis.
• Autonomy: Human-supervised. The AI automatically performed initial biometric verification, but the platform founder and administrators maintained final manual control.
• Novelty: Established threat. Demographic bias in facial recognition and invasive corporate data collection policies are well-documented, existing risks.

Impact by dimension

• Physical security: Negligible. No physical systems, kinetic weapons, or critical infrastructure were targeted or impacted by this commercial application.
• Information security: Negligible. No compromise of state intelligence capabilities, classified models, or state-sponsored disinformation campaigns occurred.
• Sovereignty: Negligible. The incident involved a private commercial social networking application and did not disrupt government operations or state authority.
• Economic security: Negligible. No strategic industries, financial markets, or critical supply chains were threatened by this incident.
• Societal stability: Minor. The app used biased biometric screening that discriminated against trans women and women of color, alongside invasive data collection, but the impact was limited to commercial app users.