Alleged Issues with Proctorio's Remote-Testing AI Prompted Suspension by University

Proctorio's remote-testing software were reported by students at the University of Illinois Urbana-Champaign for issues regarding privacy, accessibility, differential performance on darker-skinned students.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The proctoring software's facial detection algorithm performed significantly worse for Black students, failing to recognize them over half the time.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The software monitored students' private homes, webcams, and keystrokes, violating their expectation of privacy.
• 7.3 Lack of capability or robustness: The system failed to perform robustly under varying conditions, flagging students with glasses, headscarves, or disabilities.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The algorithmic bias and accessibility failures were unintentional outcomes of the deployed AI system's facial detection capabilities.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The proctoring software is an educational system affecting access to education, as it has the power to accuse students of cheating and flag them to instructors, which is classified as a High Risk AI system under the EU AI Act.

AI system and alleged parties

• AI system: Proctorio
• AI purpose: Cheating Detection; Smart Surveillance
• Behaviour type: Assistant
• Alleged developer: Proctorio
• Alleged deployer: University of Illinois
• Alleged harmed parties: University of Illinois students of color, University of Illinois students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Negligible
• Civil rights: direct Substantial, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the reports describe financial costs to academic institutions and legal expenses incurred by individuals sued by Proctorio.

Indirectly caused: Linkletter and Johnson incurred tens of thousands of dollars in legal fees defending against Proctorio's copyright lawsuits.

Inferred additional harm: Universities spent significant funds on licensing fees for emergency proctoring contracts that were later terminated.

Differential treatment

Reported: Yes, the reports explicitly describe differential treatment of Black students, students of color, and students with disabilities.

Directly caused: The facial detection software failed to recognize Black faces over half the time, requiring students of color to take extreme measures to be detected, and flagged students with headscarves or those who stim due to disabilities.

Inferred additional harm: Many other minority and disabled students likely faced unfair flags and academic hurdles due to the software's biased algorithms.

Civil rights

Reported: Yes, the reports describe concerns regarding the violation of students' civil rights, privacy, and freedom of expression.

Directly caused: The software violated students' right to privacy by recording their private homes, webcams, and keystrokes without meaningful alternatives.

Indirectly caused: Proctorio used DMCA takedowns and lawsuits to suppress the free speech and academic freedom of critics like Linkletter and Johnson.

Inferred additional harm: Widespread deployment of such surveillance tools likely chilled free expression and normalized invasive monitoring among student populations.

Privacy

Reported: Yes, the reports explicitly describe extensive privacy violations from the software's surveillance capabilities.

Directly caused: The software monitored students via webcams, microphones, browsers, and keystrokes, and recorded their private homes.

Inferred additional harm: The personal data and room recordings of thousands of students were collected and stored, posing ongoing privacy risks.

Psychological

Reported: Yes, the reports explicitly describe high levels of anxiety, stress, and panic among students forced to use the surveillance software.

Directly caused: Students reported increased test anxiety and stress due to feeling constantly surveilled and monitored by the AI software.

Inferred additional harm: It is likely that thousands of students experienced heightened academic anxiety and distress due to the invasive monitoring and fear of being falsely flagged.

People affected

• Occurrences reported: 1
• People reportedly harmed: 5000
• People reportedly exposed: 100000

Potential causes

Management

• Suppression of Critics: Filing SLAPP lawsuits against researchers who highlight software flaws.
• Neglecting Equity and Access: Deploying software without assessing impact on marginalized students.
• Lack of Operational Transparency: Management failed to provide clear details on how the software operates.

Technology

• Flawed Facial Detection Code: Relies on open-source OpenCV which has a history of racial bias.
• Inflexible Algorithmic Rules: Flags normal behaviors like stimming or wearing glasses as suspicious.
• Cognitive Bias in Tracking: Gaze tracking flags ADHD behaviors as potential cheating.

Data Inputs

• Biased Training Data: Algorithms perform poorly on diverse skin tones due to biased training sets.
• Environmental Variables: Poor lighting and webcam quality degrade facial detection performance.
• Hardware Requirements: Requires high bandwidth and webcams, disadvantaging some students.

Human Factors

• High Student Test Anxiety: Constant monitoring and invasive surveillance cause severe psychological stress.
• Unfair Living Conditions: Assumes all students have access to quiet, private testing environments.
• Over-reliance on AI Flags: Instructors may trust automated flags without conducting proper manual reviews.

Process and Methods

• Proprietary Black Box Software: Lack of source code auditing prevents independent verification of fairness.
• Inadequate Feedback Loop: Falsely flagged students face difficult processes to clear their names.
• Automatic Suspect Flagging: System automatically flags students to instructors based on arbitrary signals.

Regulatory Environment

• Abuse of DMCA Provisions: Company used copyright takedowns to silence critical security researchers.
• Lack of AI Proctoring Standards: No clear regulatory oversight governing the use of AI in education.
• Lack of Privacy Protections: Inadequate legal safeguards for student data privacy in remote settings.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts from multiple reputable sources regarding the software's functionality, the specific biases identified, and the resulting student backlash and university contract terminations.
• Ambiguities identified: The exact technical specifications of Proctorio's integration of OpenCV are not fully detailed, and Proctorio disputed the claims of racial bias, attributing failures to lighting and webcam quality.
• Alternative interpretations: Proctorio argues the software is a neutral tool and that any detection failures are due to environmental factors like lighting rather than algorithmic bias.

An AI-based remote proctoring software, Proctorio, exhibited algorithmic bias against students of color and violated student privacy during remote exams. While causing significant public backlash and academic disruption, the incident has negligible direct national security implications, representing a localized civil rights and consumer privacy concern managed through institutional policy and legal channels.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing societal and technological concern regarding algorithmic bias rather than an active, imminent national security crisis.
• Autonomy: Human-supervised. The AI system flags suspicious behavior autonomously, but human instructors review the flags to make final decisions.
• Novelty: Established threat. Algorithmic bias in facial detection and privacy concerns with commercial surveillance software are well-documented issues.

Impact by dimension

• Physical security: Negligible. No physical security threats, critical infrastructure compromise, or kinetic actions are associated with this academic software deployment.
• Information security: Negligible. The incident involves commercial exam surveillance software, with no indications of state-sponsored information operations or intelligence compromise.
• Sovereignty: Negligible. The incident affects university administrative decisions and student testing, posing no threat to state authority, elections, or core government functions.
• Economic security: Negligible. While universities terminated contracts and individuals faced legal fees, there was no threat to strategic industries, financial systems, or national technological advantages.
• Societal stability: Minor. The incident involved widespread civil rights concerns, racial bias, and privacy violations affecting thousands of students, but remained managed within standard academic and legal procedures.