Texas A&M Professor Misused ChatGPT to Detect AI Text Generation in Student Submissions

A Texas A&M-Commerce professor reportedly informed his class of his misuse of ChatGPT to detect whether student submissions had been generated by the chatbot itself, which informed their graduation status.

Risk classification

• Primary risk domain: 5 Human-Computer Interaction
• Primary risk subdomain: 5.1 Overreliance and unsafe use

The professor placed inappropriate trust and reliance on ChatGPT to perform academic plagiarism detection, a critical task the system was never designed or fit to carry out.

Additional risk subdomains

• 7.3 Lack of capability or robustness: ChatGPT failed to perform reliably by hallucinating and falsely claiming authorship of the students' essays and other unrelated texts.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Post-deployment

The incident was primarily caused by the professor's human decision to misuse ChatGPT as an AI detector, which led to unintended academic distress and false accusations.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2 (High Risk): The AI system was used in an educational setting to evaluate students and make decisions that directly affected their academic progression and graduation.

AI system and alleged parties

• AI system: None named
• AI purpose: Chatbot; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: Jared Mumm
• Alleged harmed parties: Texas A&M University students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Psychological

Reported: The report explicitly describes psychological distress and panic experienced by the accused students.

Indirectly caused: Students experienced severe distress, anxiety, and panic after being falsely accused of cheating and having their graduation diplomas temporarily withheld.

Inferred additional harm: It is highly likely that other students in the 'several' classes the professor allegedly failed experienced similar academic anxiety and distress.

People affected

• Occurrences reported: 1
• People reportedly harmed: 30
• People reportedly exposed: 30

Potential causes

Management

• Inadequate Faculty Training: University failed to train educators on AI capabilities and limitations.
• Delayed Administrative Response: Initial lack of clear guidance from administrators when students appealed.

Technology

• ChatGPT False Affirmation: ChatGPT falsely claimed authorship of the students' submitted essays.
• Lack of Built-in Detection: ChatGPT is not designed or capable of detecting AI-generated text.

Data Inputs

• Unverified Bot Responses: Professor relied on unverified ChatGPT outputs as proof of cheating.
• Ignored Google Docs Timestamps: Professor initially ignored student-provided Google Docs timestamp evidence.

Human Factors

• Professor Lack of AI Literacy: Professor did not understand how ChatGPT works, misnaming it Chat GTP.
• Confirmation Bias: Professor assumed guilt and accepted chatbot claims without verification.
• Student Distress and Panic: Students experienced severe distress due to false accusations of cheating.

Process and Methods

• Flawed AI Detection Method: Professor copied essays directly into ChatGPT to ask if it wrote them.
• Lack of Standardized Review: No formal process existed for verifying academic dishonesty involving AI.

Regulatory Environment

• Absence of AI Classroom Policies: The university lacked clear regulations and guidelines on AI use.

Information quality

• Classification confidence: High
• Reason for confidence: The incident is documented across multiple reputable news sources with consistent facts, including statements from the university spokesperson confirming the key details of the grading dispute and the professor's actions.
• Ambiguities identified: The exact number of students in the affected classes is not explicitly stated.
• Alternative interpretations: None. The facts of the professor's misuse of the tool and the subsequent resolution are clear.

A university professor falsely accused students of cheating by incorrectly relying on ChatGPT to detect AI-generated content. The incident represents a local human-computer interaction and academic policy failure with negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. The incident poses no active or developing national security crisis, representing only a localized educational policy concern.
• Autonomy: Human-controlled. The AI acted as an assistant, while the human instructor made the final grading and administrative decisions.
• Novelty: Established threat. User misunderstanding of LLM capabilities and subsequent hallucinations are well-documented and established issues.

Impact by dimension

• Physical security: Negligible. The incident was a localized academic grading dispute with no impact on physical systems, critical infrastructure, or human safety.
• Information security: Negligible. There was no compromise of intelligence capabilities, classified models, or systematic, state-sponsored information manipulation.
• Sovereignty: Negligible. The incident occurred within a university setting and did not threaten state authority, elections, or core government operations.
• Economic security: Negligible. No strategic technology was stolen, and there was no threat to national financial systems or economic stability.
• Societal stability: Negligible. While students experienced temporary academic distress, the incident was a localized administrative issue with no systemic threat to civil liberties or societal stability.