Facial Recognition Researchers Used YouTube Videos of Transgender People without Consent

YouTube videos of transgender people used by researchers to study facial recognition during gender transitions were used and distributed without permission.

Risk classification

• Primary risk domain: 2 Privacy & Security
• Primary risk subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information

The primary incident involves researchers compiling and distributing highly sensitive personal transition videos of transgender individuals without consent, and leaving them in an unprotected Dropbox.

Additional risk subdomains

• 1.3 Unequal performance across groups: Mitek's identity verification system failed to correctly process non-binary gender markers, resulting in unequal system performance and exclusion for non-binary individuals.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Post-deployment

The risks and harms were caused by human decisions, including researchers scraping data without consent and developers introducing a coding error in the ID scanning software.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes facial recognition and biometric identity verification systems, which are classified as high-risk under the EU AI Act due to their use in law enforcement, biometrics, and access to essential private services like banking.

AI system and alleged parties

• AI system: facial recognition software
• AI purpose: Face Recognition; Identification
• Behaviour type: Tool
• Alleged developer: University of North Carolina Wilmington, Karl Ricanek, Gayathri Mahalingam
• Alleged deployer: University of North Carolina Wilmington, Karl Ricanek, Gayathri Mahalingam
• Alleged harmed parties: Transgender YouTubers, transgender people

Harm severity

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Minor
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the report describes differential treatment where non-binary individuals faced authentication failures.

Directly caused: Non-binary individuals with California IDs containing an 'X' gender marker were unable to authenticate their identities through Mitek's system.

Indirectly caused: Affected individuals were forced to use alternative, more burdensome in-person verification channels to access basic services like banking.

Inferred additional harm: It is highly likely that numerous non-binary individuals in California faced systemic delays or exclusion from online financial services due to this technical discrepancy.

Civil rights

Reported: Yes, the report describes violations of privacy rights and potential civil rights implications for marginalized groups.

Directly caused: The unauthorized scraping, storage, and distribution of personal transition videos violated the basic privacy rights of the 38 individuals.

Indirectly caused: N/A

Inferred additional harm: N/A

Privacy

Reported: Yes, the report explicitly describes privacy violations concerning the unauthorized collection and distribution of personal transition videos.

Directly caused: Researchers compiled 1 million still images of 38 transgender individuals from YouTube without consent and stored them in an unprotected Dropbox.

Indirectly caused: The dataset was distributed to 15 academic institutions across seven countries, exposing the private data to external researchers without oversight.

Inferred additional harm: Additional privacy violations may have occurred as the dataset was shared further among doctoral students and other researchers without institutional review board approval.

Psychological

Reported: Yes, the report describes feelings of privacy violation and fear of targeting within the transgender community.

Directly caused: The 38 individuals in the dataset experienced distress and a sense of violation upon discovering their personal transition journeys were used without consent.

Indirectly caused: The broader transgender community experienced heightened anxiety and fear of being targeted or 'spotted' by surveillance systems trained on such data.

Inferred additional harm: N/A

People affected

• Occurrences reported: 2
• People reportedly harmed: 38
• People reportedly exposed: 38

Potential causes

Management

• Inadequate research oversight: University failed to monitor the creation and sharing of the dataset.
• Unmonitored government funding: Federal agencies funded biometric research without evaluating ethical risks.

Technology

• Algorithmic fragility to HRT: Facial recognition systems struggle to adapt to HRT facial changes.
• ID scanning software errors: Mitek systems failed to authenticate non-binary IDs due to coding errors.

Data Inputs

• Scraping without consent: YouTube videos were scraped for images without the creators' permission.
• Insecure dataset storage: The dataset was stored in an unprotected Dropbox folder accessible via URL.
• License agreement violations: Standard YouTube licenses prohibiting redistribution were ignored.

Human Factors

• Disregard for participant privacy: Researchers prioritized technical challenges over participant privacy.
• Transphobic research assumptions: Hypothesized that trans people use HRT to evade border surveillance.

Process and Methods

• Bypassing ethical review boards: The researcher did not seek institutional ethical approval for the dataset.
• Uncontrolled dataset distribution: The dataset was shared with 15 institutions globally without oversight.
• Inadequate contact for consent: Researchers failed to contact or obtain consent from all participants.

Regulatory Environment

• Lack of public data regulations: Absence of clear regulations on scraping public social media for AI training.
• Permissive default licensing: Default CC licenses allowed massive scraping of personal images.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide detailed, consistent accounts of the UNC Wilmington dataset's creation, distribution, and the privacy concerns raised by participants and researchers. The Mitek incident is also clearly described with specific technical details regarding the barcode scanning error.
• Ambiguities identified: There is some conflicting testimony between Professor Ricanek and the researchers regarding whether consent was obtained and how widely the dataset was distributed.
• Alternative interpretations: The Mitek incident could be viewed purely as a software database bug rather than an AI safety failure, though its impact on biometric identity verification aligns it with algorithmic bias.

The incidents involve unauthorized scraping of transgender individuals' biometric data for military/intelligence-funded research distributed internationally, alongside algorithmic bias in commercial identity verification. While raising significant ethical, privacy, and civil rights concerns, the direct national security impact remains minor and localized.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Multiple nations
• Primary target: United States
• Other affected: Six other unspecified nations
• Alleged perpetrator: University of North Carolina at Wilmington and Mitek

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding biometric data privacy and algorithmic bias rather than an active, immediate crisis.
• Autonomy: Human-controlled. The systems function as tools assisting human processes (research and identity verification) rather than acting autonomously.
• Novelty: Established threat. Unauthorized biometric data harvesting and algorithmic bias against demographic groups are well-documented, established issues in AI development.

Impact by dimension

• Physical security: Negligible. No physical threat, kinetic attacks, or critical infrastructure compromise occurred during these incidents.
• Information security: Minor. Research funded by FBI and U.S. Army on facial recognition evasion was stored in an unprotected Dropbox and distributed internationally, representing minor data control issues.
• Sovereignty: Negligible. No significant threat to state authority, territorial control, or core government decision-making processes was identified.
• Economic security: Minor. Mitek's system error temporarily prevented some individuals from opening online bank accounts, causing minor economic access issues but no systemic financial threat.
• Societal stability: Minor. Systematic privacy violations through unauthorized scraping of transition videos and algorithmic discrimination against non-binary individuals in ID verification, but impact remains localized.