AI-Generated-Text-Detection Tools Reported for High Error Rates

Models developed to detect whether text generation AI was used such as AI Text Classifier and GPTZero reportedly contained high rates of false positive and false negative, such as mistakenly flagging Shakespeare's works.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The incident stems directly from the AI text classifiers failing to perform reliably or robustly, yielding extremely low accuracy (26%) and high false-positive rates on human-written texts.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Educators are adopting and relying on these highly unreliable tools for critical grading and disciplinary decisions, creating a severe risk of inappropriate use in academic environments.
• 1.3 Unequal performance across groups: The reports note that the detection tools perform worse on non-English languages and present specific issues/biases when evaluating writings from neurodivergent individuals.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk of false accusations and academic penalties is caused by the incorrect classification decisions made by the AI text detection models after their public deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The reports describe AI systems being adopted in educational settings for grading and enforcing academic codes of conduct, which directly affects students' access to education and academic standing.

AI system and alleged parties

• AI system: ChatGPT, GPTZero, Originality AI (OpenAI, Originality.ai)
• AI purpose: Cheating Detection; Content Verification
• Behaviour type: Tool
• Alleged developer: OpenAI, Edward Tian
• Alleged deployer: OpenAI, Edward Tian
• Alleged harmed parties: Teachers, students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Minor
• Civil rights: direct Negligible, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly notes that the tools have 'issues with writings from neurodivergent people' and perform worse on non-English languages, creating risks of differential treatment.

Indirectly caused: The tools' technical limitations result in higher false-positive rates for non-native English speakers and neurodivergent students, leading to indirect discrimination in academic grading.

Inferred additional harm: It is highly likely that international and neurodivergent students face disproportionately higher rates of false accusations and academic scrutiny due to these algorithmic biases.

Civil rights

Reported: The report explicitly describes the risk of students being unfairly penalized and accused of academic dishonesty, which threatens their right to fair educational evaluation.

Indirectly caused: Faculty at universities like Harvard, Yale, and URI are using these unvalidated tools to enforce academic codes, exposing students to arbitrary and unfair disciplinary actions.

Inferred additional harm: It is highly likely that numerous students have already faced unjust disciplinary hearings, grade reductions, or suspensions based on unvalidated algorithmic accusations.

Epistemic

Reported: The report describes epistemic challenges, noting that the tools generate false classifications (labeling human text as AI) and that the internet may become flooded with untrustworthy content.

Directly caused: The AI Text Classifier directly generated false classifications, labeling Shakespeare's Macbeth and Raschka's textbook as AI-generated.

Indirectly caused: The unreliability of these tools undermines the ability of educators to establish objective truth regarding the authorship of student work.

Inferred additional harm: The widespread failure of detection tools may lead to a general breakdown of trust in digital written communication and academic grading systems.

People affected

• Occurrences reported: 1
• People reportedly exposed: 10000

Potential causes

Management

• Prioritizing Speed Over Safety: Releasing imperfect tools to the public before they are reliable.
• Inadequate Risk Assessment: Failing to prevent the real-world harm of educators using faulty grading tools.

Technology

• Low Classifier Accuracy: Only 26 percent true positive rate and 9 percent false positive rate.
• Easy Detection Evasion: Users can bypass detection easily through reprompting or paraphrasing.
• Unreliability on Short Text: The classifier is highly unreliable when evaluating shorter text passages.

Data Inputs

• Training Dataset Biases: Lack of training on Old English and non-English languages causes errors.
• Spurious Style Correlations: Lack of style instructions in training data leads to neutral sentiment bias.
• Pre-2022 Data Overlap: Texts written before 2022 are misclassified because they were training data.

Human Factors

• Premature Adoption by Educators: Teachers use unreliable tools to grade and police academic dishonesty.
• Overreliance on Automated Labels: Users blindly trust predicted labels instead of critically analyzing scores.
• Fear of Student Cheating: Panic over ChatGPT cheating drives immediate adoption of unverified tools.

Process and Methods

• Insufficient Tool Validation: Releasing a tool publicly with only 26 percent accuracy to gather feedback.
• Lack of Performance Transparency: No confusion matrix or paper shared to explain false positive rates.
• Inadequate Complementary Methods: Deploying the tool without establishing reliable alternative verification.

Regulatory Environment

• Lack of Industry Regulation: The generative AI detection space operates like a Wild West without rules.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed statistics (26% accuracy, 9% false positives) and specific examples of the tool's failure (Macbeth, Raschka's book). The implications for educators and students are clearly articulated across multiple sources.
• Ambiguities identified: The exact number of students who have already been falsely penalized is not quantified, only the fact that universities are adopting the tools.
• Alternative interpretations: None. The failure of the text classifiers is unambiguous.

The widespread deployment of unreliable AI text detection tools has resulted in high false-positive rates, disproportionately affecting non-native English speakers and neurodivergent individuals. While this poses real-world challenges for academic integrity and student welfare, it represents a minor societal concern manageable through standard institutional procedures, with negligible implications for core national security.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global educational institutions
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Refers to an ongoing, long-term challenge regarding the reliability of AI detection tools rather than an active, imminent national security crisis.
• Autonomy: Human-controlled. The tools function as assistive classifiers, leaving final disciplinary and grading decisions to human educators.
• Novelty: Evolved capability. Represents an evolved capability in response to the rapid rise of LLM-generated content, adapting older text-classification techniques to new models.

Impact by dimension

• Physical security: Negligible. No physical security threats, kinetic attacks, or critical infrastructure impacts are associated with these text classification tools.
• Information security: Negligible. No evidence of state-sponsored information warfare, intelligence compromise, or classified data theft related to these academic tools.
• Sovereignty: Negligible. No direct threat to state authority, electoral systems, or core government decision-making processes.
• Economic security: Negligible. While highlighting limitations in AI verification technology, the incident does not threaten strategic industries, financial systems, or national technological advantage.
• Societal stability: Minor. The tool's high false-positive rates and performance biases against non-native English speakers and neurodivergent individuals present minor risks of unfair treatment, manageable within standard academic procedures.