ProctorU’s Identity Verification and Exam Monitoring Systems Provided Allegedly Discriminatory Experiences for BIPOC Students

An exam monitoring service used by the University of Toronto was alleged by its students to have provided discriminatory check-in experiences via its facial recognition's failure to verify passport photo, disproportionately enhancing disadvantaging stress level for BIPOC students.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The remote proctoring software exhibited unequal performance across demographic groups, specifically failing to recognize BIPOC students' faces and passports compared to white students.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The facial recognition technology lacked the robustness to perform reliably across different skin tones and lighting conditions.
• 5.1 Overreliance and unsafe use: Educational institutions overrelied on automated proctoring tools for high-stakes testing despite known equity and performance limitations.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the ProctorU AI system's biased facial recognition algorithm failing to verify BIPOC students post-deployment, which was an unintentional outcome of its design.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2: High Risk. The system is an educational assessment tool used in higher education, which falls under High Risk as it affects students' access to education. The report notes that the software's inability to recognize facial features 'temporarily barred them from accessing the software' and taking their exams.

AI system and alleged parties

• AI system: Examity, ProctorU
• AI purpose: Face Recognition; Cheating Detection
• Behaviour type: Assistant
• Alleged developer: ProctorU
• Alleged deployer: University of Toronto
• Alleged harmed parties: University of Toronto BIPOC students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the report explicitly describes differential treatment of BIPOC students compared to white students.

Directly caused: BIPOC students faced friction, identity verification failures, and delays (e.g., losing 5 out of 15 minutes of check-in time) that white students did not face.

Inferred additional harm: It is highly likely that thousands of BIPOC students at various universities using ProctorU and Examity experienced similar discriminatory friction and delays.

Civil rights

Reported: Yes, the report explicitly mentions concerns regarding equity and discriminatory biases raised by civil rights attorneys and US Senators.

Directly caused: BIPOC students and students wearing religious dress were temporarily barred from accessing the software, violating their right to equal access to education.

Inferred additional harm: Systemic deployment of biased facial recognition in high-stakes testing likely violated the civil rights of numerous minority students across multiple institutions.

Privacy

Reported: Yes, the report mentions that US Senators raised concerns about student privacy in relation to exam monitoring services.

Directly caused: Students were subjected to 360-degree room scans, proof of inactive phones, and full computer sweeps to take exams.

Inferred additional harm: Thousands of students using these platforms had their personal spaces and computer data exposed to third-party proctoring companies, raising significant privacy risks.

Psychological

Reported: Yes, the report explicitly describes psychological distress and anxiety caused to students.

Directly caused: Students like Chelsea Okankwu and Maame Adjoa experienced significant pre-exam stress, anxiety, and feeling disadvantaged due to identity verification failures.

Inferred additional harm: It is likely that hundreds of other BIPOC students experienced similar pre-exam anxiety and stress due to verification failures, though specific numbers are not detailed.

People affected

• Occurrences reported: 1
• People reportedly harmed: 22
• People reportedly exposed: 100

Potential causes

Management

• Decentralized Decision-making: Individual faculties decide on software usage without central oversight.
• Continued Use of Biased Vendors: Partnerships with ProctorU and Examity continued despite known software biases.

Technology

• Bias in Facial Recognition Tech: Software struggles to identify darker skin tones, causing check-in failures.
• Automated Proctoring Failures: AI systems fail to recognize passports or facial features of BIPOC students.

Data Inputs

• Underrepresentative Training Data: Datasets used to train AI models underrepresent BIPOC individuals.

Human Factors

• Increased Student Stress: Verification failures and intense check-in requirements cause pre-exam anxiety.

Process and Methods

• Intrusive Check-in Procedures: Requirements like 360 room scans and computer sweeps enhance test anxiety.
• Lack of Alternative Assessments: Heavy reliance on high-stakes exams instead of project-based learning.

Regulatory Environment

• Lack of University-wide Policy: UofT has no centralized regulation banning or strictly limiting e-proctoring.
• Non-binding Equity Guidelines: Official advice to limit proctoring is framed as a recommendation, not a rule.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, first-hand accounts from affected students (Chelsea Okankwu, Maame Adjoa) and corroborate these with statements from university officials, civil rights attorneys, and US Senators. The technical failure (facial recognition bias against BIPOC individuals) is well-documented and aligned with established AI research.
• Ambiguities identified: There is some contradiction between ProctorU's claim that they are not an 'automated proctoring service' and their technical documentation which supports 'Automated Proctoring' for Chromebook users.

Remote proctoring software ProctorU exhibited racial bias, failing to verify the identities of BIPOC students at North American universities. While raising important civil rights and ethical concerns regarding AI deployment in education, the incident has negligible direct implications for national security.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: Canada
• Other affected: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing, systemic challenge with algorithmic bias rather than an active national security crisis.
• Autonomy: Human-supervised. The AI system flagged identity verification issues, but human proctors were involved in making the final decisions regarding student access.
• Novelty: Established threat. Demographic bias in facial recognition technology is a well-documented and established technical limitation.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic targeting systems were impacted or involved in this incident.
• Information security: Negligible. No intelligence capabilities were compromised, and there was no evidence of systematic foreign information warfare or espionage.
• Sovereignty: Negligible. The incident involved academic software used in higher education and did not threaten state authority, elections, or core government operations.
• Economic security: Negligible. While causing stress to students, the incident did not threaten strategic industries, financial stability, or national technological advantages.
• Societal stability: Minor. The software exhibited racial bias affecting BIPOC students, raising civil rights concerns, but the impact was limited to academic settings and manageable locally.