Teenager at Broward College Allegedly Wrongfully Accused of Cheating via Remote Proctoring

Broward College’s use of remote proctoring system and reliance on its flagging algorithm allegedly led to a wrongful accusation of academic dishonesty in a biology exam of a Florida teenager.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The remote proctoring system utilizes Amazon Rekognition, which has documented lower accuracy and higher bias when analyzing darker-skinned females, directly impacting the Black female student who was also misgendered by the tool.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Educators overrelied on the automated flags generated by the AI system, treating them as definitive proof of cheating without proper critical evaluation.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was initiated by the Honorlock AI system's automated flagging of benign human behavior, which was an unexpected outcome of its deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2 (High Risk): The AI system is used in an educational setting ('remote proctoring companies offered web browser extensions... to prevent the temptation to cheat') where its outputs directly affect student evaluation and academic standing, which falls under educational systems affecting access to education.

AI system and alleged parties

• AI system: Honorlock remote proctoring system, Rekognition (Amazon, Honorlock)
• AI purpose: Cheating Detection; Face Recognition
• Behaviour type: Tool
• Alleged developer: Honorlock
• Alleged deployer: Broward College
• Alleged harmed parties: unnamed Florida teenager

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of the student based on algorithmic flagging.

Directly caused: The Black female student was flagged by a facial detection system (Amazon Rekognition) known to have biased performance and lower accuracy for darker-skinned females.

Indirectly caused: The student received a failing grade of zero and an academic warning on her record, while other students who did not trigger the algorithm faced no such penalties.

Inferred additional harm: Minority students are likely subjected to higher rates of false accusations and academic penalties due to systemic demographic biases in facial detection algorithms.

Privacy

Reported: The report explicitly describes privacy concerns related to the collection of sensitive student data.

Directly caused: The Honorlock system collected the student's biometric data, room scans, screen recordings, audio, and keystrokes.

Inferred additional harm: Millions of students are subjected to invasive surveillance and have their sensitive biometric and personal data harvested by private proctoring companies.

Psychological

Reported: The report explicitly describes psychological distress experienced by the student.

Directly caused: The 17-year-old student was left distraught and anxious, stating she now tries to become like a mannequin during tests out of fear of being flagged again.

Inferred additional harm: It is highly likely that other students falsely accused of cheating by automated proctoring systems experience similar academic anxiety and psychological distress.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Shift of Responsibility to Schools: Honorlock delegates the challenging task of interpreting flags to educators.
• Prioritizing Automation over Review: Schools adopted fully automated proctoring without hiring human proctors.

Technology

• Ambiguous Face Detection Flags: Rekognition flags normal behaviors like looking down as suspicious.
• Potential Algorithmic Bias: Face detection tools perform less reliably on darker-skinned females.
• False Face Detection Triggers: Software historically flagged faces in posters or photos as extra people.

Data Inputs

• Lack of Contextual Video Data: The algorithm only analyzes visual cues without understanding human behavior.
• Biased Training Datasets: Underlying facial models perform poorly on diverse skin tones and genders.

Human Factors

• Confirmation Bias by Educators: Teachers treated algorithmic flags as definitive proof of academic dishonesty.
• Normal Human Movements Penalized: Students looking down to think are falsely categorized as cheating.
• Lack of Technical Literacy: Educators lack training to properly evaluate flagged automated alerts.

Process and Methods

• Inadequate Video Review Process: Faculty did not properly investigate the context of the 50-second clip.
• Over-reliance on Automated Alerts: Schools used automated flags directly to issue disciplinary actions.
• Asynchronous Course Distance: Fully remote and asynchronous classes reduced student-teacher interaction.

Regulatory Environment

• Absence of Student Privacy Laws: Lack of regulations governing invasive remote proctoring surveillance.
• No Algorithmic Audit Standards: No regulatory standards require independent bias audits for proctoring tools.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides a highly detailed account of a specific student's experience, backed by direct quotes, a review of the video evidence, and technical details regarding the underlying AI model (Amazon Rekognition) and its documented biases.
• Ambiguities identified: The exact extent to which the algorithm's performance was degraded by the student's race or gender in this specific instance remains unproven, though the general bias of the tool is well-documented.
• Alternative interpretations: The school framed the issue as a failure to comply with test-taking directions (looking away) rather than an algorithmic error, though the automated flag was the sole trigger for the investigation.

A Florida teenager was falsely accused of cheating by Honorlock proctoring software using Amazon Rekognition, highlighting ongoing concerns about demographic bias and privacy in commercial AI. While raising important civil rights and ethical questions, the incident has negligible direct implications for national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident represents an ongoing societal and technological challenge regarding algorithmic bias rather than an active national security crisis.
• Autonomy: Human-supervised. The AI system autonomously flagged the student, but human administrators reviewed the flag and made the final decision to uphold the accusation.
• Novelty: Established threat. Demographic biases in facial recognition models and false positives in remote proctoring software are well-documented, established issues.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic threats were involved in this academic proctoring incident.
• Information security: Negligible. No intelligence capabilities, classified models, or state-sponsored information warfare operations were involved.
• Sovereignty: Negligible. The incident involves a domestic educational institution and does not impact state authority, border control, or core government operations.
• Economic security: Negligible. The incident represents an educational dispute and does not threaten strategic industries, national economic stability, or critical supply chains.
• Societal stability: Minor. Highlights concerns regarding algorithmic bias and privacy in widely deployed commercial AI, but impact is localized and manageable through standard civil and administrative procedures.