Students Allegedly Wrongfully Accused of Cheating via Medical School's Internal Software

Dartmouth's Geisel School of Medicine allegedly falsely accused students of cheating during remote exams using an internally built system which tracked student activity patterns without their knowledge on its learning management platform.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The Canvas system failed to perform reliably by automatically generating false user activity data, leading to erroneous accusations of cheating.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: School administrators overrelied on Canvas data as a forensic tool to prove cheating, despite the software not being designed for that purpose.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk was caused by the Canvas system automatically generating false activity data, which was an unexpected outcome of the system's normal operations.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is an educational monitoring tool used to detect cheating, directly affecting students' access to education and vocational training.

AI system and alleged parties

• AI system: Canvas
• AI purpose: Cheating Detection; Activity Tracking
• Behaviour type: Tool
• Alleged developer: Geisel School of Medicine's Technology staff, Canvas
• Alleged deployer: Geisel School of Medicine
• Alleged harmed parties: Sirey Zhang, Geisel School of Medicine's students, Geisel School of Medicine's professors, Geisel School of Medicine's accused students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Minor
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the report describes students facing suspensions and being required to repeat a year of school at a cost of nearly $70,000.

Indirectly caused: Average financial loss per occurrence: $70000 for students forced to repeat a year of medical school due to the false accusations.

Civil rights

Reported: Yes, the report describes complaints of unfair treatment, lack of due process, and invasive surveillance.

Indirectly caused: At least 17 students had their right to a fair disciplinary process compromised, receiving less than 48 hours to respond and minimal time to present their cases.

Privacy

Reported: Yes, the report describes retroactive tracking of student online activity without their knowledge.

Indirectly caused: Dartmouth used Canvas to retroactively track the online activity of students during exams without their prior knowledge.

Psychological

Reported: Yes, the report explicitly describes severe emotional distress, isolation, and trauma experienced by the accused students.

Indirectly caused: At least 17 students experienced severe anxiety, isolation, and trauma due to being falsely accused of honor code violations that threatened their medical careers.

People affected

• Occurrences reported: 1
• People reportedly harmed: 17
• People reportedly exposed: 38

Potential causes

Management

• Rushing Accusations: Administrators rushed to accuse students without verifying data accuracy.
• Lack of Technical Expertise: School lacked the expertise to understand the limitations of Canvas data.
• Inadequate Initial Guidelines: School failed to instruct students to log out of Canvas before exams.

Technology

• Canvas Not Designed for Forensics: Canvas is an LMS, not a forensic tool for detecting cheating.
• Automated Data Generation: Canvas automatically generates activity data even when devices are idle.
• Flawed Pattern Detection: The custom audit system misidentified background syncs as human activity.

Data Inputs

• Background Page Activity: Open Canvas tabs generated false-positive activity logs during exams.
• Incomplete Activity Logs: Students were not provided with full logs to verify the data's context.
• Lack of Question Context: Logs lacked the wording of exam questions to verify relevance of pages.

Human Factors

• Failure to Log Out of Canvas: Students left Canvas pages open on backup devices during exams.
• Overreliance on Tech Evidence: Administrators assumed automated logs were infallible proof of cheating.
• Pressure to Plead Guilty: Students were advised to admit guilt despite maintaining their innocence.

Process and Methods

• Inadequate Defense Window: Students had less than 48 hours to respond to complex technical charges.
• Lack of Algorithmic Validation: The custom pattern-matching algorithm was not properly validated first.
• Flawed Disciplinary Hearings: Students were given only two minutes to present their cases online.

Regulatory Environment

• Lack of Precedents and Standards: No established standards exist to prevent accusations on flimsy tech evidence.
• Inadequate Privacy Regulation: Pandemic accelerated adoption of invasive tech without privacy protections.
• Lack of Oversight on EdTech: No external regulatory oversight on the accuracy of exam monitoring tools.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides highly detailed, credible information from a reputable source, including technical analysis of the software code and direct statements from both students and administrators.
• Ambiguities identified: None of significance.
• Alternative interpretations: None.

An internal data-mining system at Dartmouth falsely flagged 17 medical students for cheating due to automated background processes. The incident resulted in localized academic penalties and distress but carries negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Dartmouth Geisel School of Medicine

Threat characteristics

• Imminence: Long-term. This was a localized academic incident that has been resolved, presenting no active or imminent threat to national security.
• Autonomy: Human-supervised. The data-mining system automatically flagged activity, but human administrators reviewed the logs and made the final disciplinary decisions.
• Novelty: Established threat. False positives and errors from automated monitoring and logging software are well-established technological issues.

Impact by dimension

• Physical security: Negligible. The incident involved academic monitoring software and had no impact on physical systems, critical infrastructure, or human safety.
• Information security: Negligible. There was no compromise of intelligence capabilities, classified data, or state-sponsored information operations.
• Sovereignty: Negligible. The event was strictly an internal academic disciplinary matter at a private university with no threat to state authority or government operations.
• Economic security: Negligible. While individual students faced financial losses, there was no threat to national economic stability, strategic industries, or technological competitive advantage.
• Societal stability: Negligible. The incident caused localized distress and due process concerns for 17 students, but posed no large-scale threat to societal stability, civil liberties, or national human rights.