Turnitin's ChatGPT-Detection Tool Falsely Flagged Student Essays as AI-Generated

Turnitin's tool to detect writing generated by ChatGPT was reported for incorrectly flagging high school students' original essays as AI-generated, accusations of which are argued as reinforcement of bias from teachers due to the inability to compare against source documents.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The Turnitin AI detector fails to perform reliably under varying conditions, resulting in false positives that incorrectly flag human-written essays as AI-generated.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Educators may overrely on the detector's scores, treating them as definitive proof of cheating despite the system's known error rates.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The false positive flags are an unexpected and unintentional outcome of the AI detector's statistical analysis after it was deployed to schools.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is an educational tool used to assess student work and detect plagiarism, which directly affects students' access to education and academic standing.

AI system and alleged parties

• AI system: ChatGPT, Turnitin AI-writing detector (OpenAI, Turnitin)
• AI purpose: Cheating Detection; Content Verification
• Behaviour type: Tool
• Alleged developer: Turnitin
• Alleged deployer: Turnitin
• Alleged harmed parties: Lucy Goetz, high school students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the report notes the potential for teacher bias to creep in and the risk of unfair treatment of students.

Indirectly caused: The report highlights that false positives leave the door open for teacher bias to creep in, potentially leading to unequal treatment of students based on subjective relationships.

Inferred additional harm: Students writing in formulaic styles or those using English as a second language may face disproportionate false positives and subsequent unfair academic scrutiny.

Civil rights

Reported: Yes, the report describes the risk of baseless academic-integrity investigations violating students' rights to fair evaluation.

Indirectly caused: The deployment of the unvetted detector created a risk of students being subjected to baseless academic-integrity investigations.

Inferred additional harm: Thousands of students could potentially face unjust disciplinary actions, suspension, or damage to their academic records due to false positive flags.

Psychological

Reported: Yes, the report describes the psychological distress and fear experienced by students facing potential false accusations of cheating.

Indirectly caused: The student Goetz described the prospect of being accused of AI cheating as 'especially scary' due to the difficulty of proving innocence.

Inferred additional harm: It is highly likely that many other students falsely flagged by the system experienced significant anxiety, stress, and fear of academic ruin.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 5

Potential causes

Management

• Prioritization of Market Share: Releasing tool quickly due to fear of plagiarism software obsolescence.
• Premature Deployment Decisions: Activating system for millions of students despite known accuracy issues.
• Deficient Risk Assessment: Management failed to address academic impacts of false positive accusations.

Technology

• Statistical Detection Limitations: Detectors rely on average probability, misidentifying standard human writing.
• Rapid AI Capabilities Evolution: New models like GPT-4 bypass detectors tuned for older models.
• Vulnerability to Paraphrasing: Paraphrasing tools like Quillbot easily bypass detection algorithms.

Data Inputs

• Lack of Source Documents: AI generation leaves no source files to verify cheating, unlike plagiarism.
• Stylistic Writing Constraints: Essays in economics or math use set styles, mimicking AI average patterns.
• Multilingual Translation Gaps: Translated text from ChatGPT is not flagged, bypassing detector inputs.

Human Factors

• Overreliance on Software Scores: Teachers may treat probability scores as definitive proof of cheating.
• Susceptibility to Teacher Bias: Lack of physical proof allows subjective bias to influence accusations.
• High Anxiety of Accused Students: Students face extreme distress due to unprovable false accusations of cheating.

Process and Methods

• Insufficient Product Vetting: Software launched to millions of users without widespread independent vetting.
• Tuning Tradeoff Failures: Tightening thresholds to reduce false positives increased false negatives.
• Ambiguous Guidance for Educators: Marketing claims high accuracy while disclaimers warn not to use it directly.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, first-hand testing results of the Turnitin detector, quotes from Turnitin executives, and perspectives from affected students and educators. The role of the AI system and the nature of its failures are explicitly documented.
• Ambiguities identified: The exact number of real-world false positives occurring post-deployment is not quantified, only estimated based on Turnitin's internal testing and the journalist's small-scale test.

Turnitin released an AI-writing detection tool that produced false positives, mistakenly flagging original student essays as AI-generated. While causing localized academic distress and highlighting risks of algorithmic overreliance, the incident has negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing software reliability and educational policy concern rather than an active national security crisis.
• Autonomy: Human-supervised. The AI tool autonomously analyzes text and outputs a score, but human educators remain responsible for interpreting results and making final disciplinary decisions.
• Novelty: Evolved capability. The use of statistical models to detect machine-generated text represents an evolved capability in the ongoing challenge of academic plagiarism detection.

Impact by dimension

• Physical security: Negligible. No physical security threat, kinetic targeting, or critical infrastructure compromise reported in relation to this academic tool.
• Information security: Negligible. The incident involves academic integrity software and does not affect national intelligence capabilities, classified models, or state-sponsored information operations.
• Sovereignty: Negligible. No threat to state authority, territorial control, electoral systems, or core government decision-making processes.
• Economic security: Negligible. No threat to strategic industries, financial systems, critical supply chains, or national technological competitive advantage.
• Societal stability: Negligible. While causing academic distress and potential unfair disciplinary actions at an institutional level, the incident has negligible impact on national-scale societal stability or civil liberties.