AI Detection Tools Allegedly Misidentify Neurodivergent and ESL Students' Work as AI-Generated in Academic Settings

AI writing detection tools have reportedly continued to falsely flag genuine student work as AI-generated, disproportionately impacting ESL and neurodivergent students. Specific cases include Moira Olmsted, Ken Sahib, and Marley Stevens, who were penalized despite writing their work independently. Such tools reportedly exhibit biases, leading to academic penalties, probation, and strained teacher-student relationships.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The AI detection systems perform with significantly lower accuracy for ESL and neurodivergent students, falsely flagging their writing as AI-generated due to their formulaic or non-native writing styles.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI detection tools fail to perform reliably, exhibiting a 1% to 4% false positive rate on human-written text.
• 5.1 Overreliance and unsafe use: Educators overrely on AI detectors to make critical disciplinary decisions, treating them as definitive proof of cheating.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The false accusations of cheating are caused by the incorrect classifications (false positives) generated by deployed AI writing detection systems, which is an unintentional failure of their design.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The report describes AI systems used in educational settings that directly affect students' grades, academic standing, and access to education, which aligns with the High Risk category for educational and vocational training systems.

AI system and alleged parties

• AI system: Copyleaks, GPTZero, Turnitin (Copyleaks, GPTZero, Turnitin)
• AI purpose: Cheating Detection; Content Verification
• Behaviour type: Tool
• Alleged developer: Turnitin, GPTZero, Copyleaks
• Alleged deployer: Universities, Colleges, Central Methodist University, Berkeley College
• Alleged harmed parties: students, Neurodivergent students, Moira Olmsted, Marley Stevens, Ken Sahib, ESL students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: Yes, the report explicitly describes differential treatment of neurodivergent and ESL students.

Directly caused: AI detectors disproportionately flag ESL students (over half in a Stanford study) and neurodivergent students due to their formulaic or mechanical writing styles.

Indirectly caused: These groups face higher rates of false accusations, leading to unfair academic penalties, zero grades, and probation compared to native, neurotypical peers.

Inferred additional harm: Systemic bias in AI detection software likely leads to widespread, undocumented academic discrimination against non-native English speakers and neurodivergent students globally.

Civil rights

Reported: Yes, the report describes students' rights to fair evaluation and education being compromised.

Directly caused: Students were given failing grades (zeros) and put on academic probation without adequate proof of misconduct, violating basic student rights to fair process.

Indirectly caused: The fear of false positives has restricted students' freedom to write naturally or use standard grammar tools like Grammarly.

Inferred additional harm: Widespread adoption of these tools without due process likely violates the educational rights of thousands of students who are falsely penalized.

Psychological

Reported: Yes, the report explicitly describes anxiety, paranoia, and distress among students.

Directly caused: Students experienced a 'punch in the gut', felt 'overwhelmed', and classrooms are 'plagued by anxiety and paranoia' over false accusations.

Indirectly caused: Students became 'obsessive' about avoiding accusations, spending hours screen-recording their work and tweaking writing to sound worse.

Inferred additional harm: Thousands of students likely suffer from academic anxiety and stress due to the fear of being falsely accused of cheating by automated tools.

Epistemic

Reported: Yes, the report describes the generation of false accusations of cheating.

Directly caused: AI detectors generate false claims (with up to 100% stated certainty) that human-written essays are AI-generated.

Indirectly caused: This has fractured trust between educators and students, creating a climate where genuine human writing is doubted.

Inferred additional harm: The erosion of trust in human authorship and the pollution of academic evaluation with false algorithmic verdicts undermines the integrity of educational assessment.

People affected

• Occurrences reported: 1
• People reportedly harmed: 13
• People reportedly exposed: 500

Potential causes

Management

• Adoption of Unverified Software: Schools purchase and deploy AI detectors without verifying their accuracy rates.
• Prioritizing Cheating Prevention: Administrators prioritize catching cheaters over the risk of false accusations.

Technology

• Inherent False Positive Rates: Detectors have a 1-4% false positive rate, leading to incorrect flags.
• Perplexity-Based Detection: Detectors flag formulaic writing styles, misidentifying human text as AI.
• Grammar Tools Triggering Flags: Using tools like Grammarly to improve text can cause 100% AI-written flags.

Data Inputs

• Lack of Neurodivergent Data: Models are not trained on neurodivergent student writing styles.
• Biased Training Data: Models struggle with non-native English writing, leading to higher false rates.

Human Factors

• Teacher Over-Reliance on Tools: Educators treat detector scores as absolute proof of cheating.
• Formulaic Student Writing Style: Neurodivergent and ESL students naturally write in a more structured manner.

Process and Methods

• Lack of Clear Disciplinary Path: Schools lack standardized processes to handle and resolve false AI accusations.
• No Verification of Writing Steps: Students lack easy ways to prove their authentic drafting and writing process.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides extensive, high-quality information including specific student case studies, independent testing data from Bloomberg, academic studies from Stanford, and statements from both AI developers and educators. The causal links between the AI tool failures and the resulting academic and psychological harms are clearly documented.
• Ambiguities identified: None significant; the mechanics of the false positives and their impact on students are well-explained.
• Alternative interpretations: One could argue the primary failure is human overreliance by teachers rather than the AI's technical limitations, but both aspects are thoroughly covered.

AI writing detection tools are generating high rates of false positives, disproportionately penalizing ESL and neurodivergent students in US universities. While this raises minor concerns regarding algorithmic discrimination and student rights, it represents negligible threat to core national security interests.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This is an ongoing systemic issue with deployed software rather than an active, fast-moving national security crisis.
• Autonomy: Human-supervised. The AI detection tools operate autonomously to flag text, but human educators ultimately review the outputs and make disciplinary decisions.
• Novelty: Established threat. Algorithmic bias and false positives in automated classification systems are well-documented, established issues in AI deployment.

Impact by dimension

• Physical security: Negligible. The incident involves text analysis tools in academic settings and poses no threat to physical systems, critical infrastructure, or human safety.
• Information security: Negligible. No compromise of intelligence capabilities, classified data, or state-sponsored information warfare operations is indicated.
• Sovereignty: Negligible. The incident is restricted to educational institutions and does not impact state authority, border control, or core government operations.
• Economic security: Negligible. While affecting individual students academically and potentially financially, the incident does not threaten strategic industries, financial systems, or national technological advantage.
• Societal stability: Minor. The high false-positive rate for ESL and neurodivergent students represents a minor civil rights and discriminatory impact within the national education system, though national security implications remain negligible.