Facial Recognition in Remote Learning Software Reportedly Failed to Recognize a Black Student’s Face

A Black student's face was not recognized by the remote-proctoring software during check-in of a lab quiz, causing her to excessively change her environments for it to work as intended.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The facial recognition software exhibits unequal performance across demographic groups, failing to recognize non-white skin tones significantly more often than white skin tones.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the AI-based facial detection system failing to recognize a student's face post-deployment, which was an unintentional outcome of the software's design.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used in an educational setting ('Testing and Recognition Software that is essential for students across the world to use for tests and take classes'), which falls under AI systems affecting access to education and vocational training.

AI system and alleged parties

• AI system: remote-proctoring software
• AI purpose: Face Recognition; Cheating Detection
• Behaviour type: Tool
• Alleged developer: unknown
• Alleged deployer: unknown
• Alleged harmed parties: Black test-takers, Black students, Amaya Ross

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment where students of color are flagged to their teachers because the testing software cannot recognize their skin tone.

Directly caused: Amaya, a Black student, was not recognized by the remote-proctoring software due to her skin tone, preventing her from easily taking her lab quiz.

Indirectly caused: Students of color are flagged to their teachers as potential cheaters or non-compliant because the software fails to detect them.

Inferred additional harm: Many other non-white students globally likely face similar unfair flagging and academic hurdles due to the biased performance of facial detection software.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Negligent Procurement: Schools acquire essential software without vetting for algorithmic bias.

Technology

• Biased Facial Detection: The software fails to detect or recognize non-white skin tones.
• Inadequate Algorithm Design: Algorithms are optimized primarily for lighter skin tones.

Data Inputs

• Unrepresentative Training Data: Training datasets lack sufficient images of people of color.

Human Factors

• Overreliance on Automated Flags: Teachers trust software alerts without verifying student presence.

Process and Methods

• Deficient Software Testing: Testing protocols failed to evaluate performance across diverse groups.

Regulatory Environment

• Lack of Bias Standards: No regulations mandate demographic bias testing for school software.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the specific failure mode (facial recognition failing on darker skin tones) and the context of use (educational testing). While the specific software name is omitted, the nature of the AI safety risk and its real-world impact are unambiguous.
• Ambiguities identified: The specific brand or developer of the proctoring software is not named.
• Alternative interpretations: None. The incident is a classic case of demographic bias in computer vision.

An incident involving facial recognition bias in remote-proctoring software highlights ongoing systemic issues with algorithmic fairness in educational tools. While representing a significant civil rights and educational equity concern, the incident carries negligible direct implications for national security.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents an ongoing systemic issue with commercial software rather than an active national security crisis requiring immediate intervention.
• Autonomy: Human-supervised. The AI system operates with human supervision, as it flags potential anomalies to teachers who make the final determination.
• Novelty: Established threat. Demographic bias and unequal performance across racial groups in facial recognition models is a well-documented and established AI safety issue.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety vectors were impacted or threatened by this educational software failure.
• Information security: Negligible. The incident does not involve intelligence compromise, classified data theft, or state-sponsored information operations.
• Sovereignty: Negligible. No threat to state authority, electoral systems, border control, or core government operations was observed.
• Economic security: Negligible. The failure of educational proctoring software does not pose a threat to strategic industries, financial systems, or national technological competitiveness.
• Societal stability: Minor. While the software exhibits systemic demographic bias affecting access to education, the national security impact is minor and manageable through civil rights and educational regulatory frameworks.