Open-Source Generative Models Abused by Students to Cheat on Assignments and Exams

Students were reportedly using open-source text generative models such as GPT-3 and ChatGPT to complete school assignments and exams such as writing reports, essays.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involves students using AI systems to gain an unfair academic advantage through plagiarism and cheating, which explicitly falls under AI-facilitated plagiarism in education within this sub-domain.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Students rely on AI to do their 'busywork' or critical thinking assignments, bypassing the learning process and risking cognitive enfeeblement.
• 7.3 Lack of capability or robustness: The AI systems generate plausible-sounding but incorrect information ('well-written wrong'), leading to epistemic errors in student submissions.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The academic disruption and cheating are caused by the intentional decisions of students to use deployed AI systems to complete their assignments without authorization.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The AI systems described are chatbots (ChatGPT) and generative text models, which fall under Limited Risk due to transparency obligations, requiring users to be aware they are interacting with AI.

AI system and alleged parties

• AI system: ChatGPT, Sudowrite (OpenAI)
• AI purpose: Writing Assistant; Question Answering
• Behaviour type: Assistant
• Alleged developer: Sudowrite, OpenAI
• Alleged deployer: students
• Alleged harmed parties: Teachers, non-cheating students, cheating students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 4497

Potential causes

Management

• Inadequate Risk Assessment: School administrations failed to assess the risks of AI before public launch.
• Lack of Educator Training: Schools did not provide teachers with training on how to handle AI tools.
• Slow Institutional Adaptation: University boards are slow to update policies, leaving teachers to cope alone.

Technology

• Generative AI Text Generation: AI produces coherent, human-like text indistinguishable from human writing.
• Imperfect AI Detection Tools: Current detection tools yield false positives and negatives, lacking certainty.
• Lack of Citation Generation: AI tools generate text without providing citations, complicating verification.

Data Inputs

• Massive Training Datasets: AI trained on vast web data, enabling it to mimic diverse academic styles.
• Plausible but False Information: AI generates statistically probable words that are factually incorrect.
• Lack of Academic Fine-Tuning: AI models are not fine-tuned for academic context, leading to factual errors.

Human Factors

• Student Desire for Shortcuts: Students use AI to complete busywork quickly and avoid academic struggle.
• Educator Unpreparedness: Teachers were blindsided by the sudden public release of capable AI chatbots.
• Acceptance of Academic Dishonesty: Some students rationalize AI cheating to gain an unfair grade advantage.

Process and Methods

• Vulnerable Assessment Methods: Take-home essays and open-book exams are easily compromised by AI tools.
• Inadequate Plagiarism Software: Traditional detectors look for exact matches and fail to flag original AI text.
• Lack of Verification Workflows: Schools lack clear workflows to handle suspected but unproven AI cheating.

Regulatory Environment

• Outdated Academic Honor Codes: Plagiarism definitions in honor codes do not explicitly cover generative AI.
• Delayed Policy Response: Education departments delayed in establishing clear guidelines on AI usage.
• Absence of Clear AI Regulations: Lack of external regulatory standards to govern generative AI in education.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed, and multi-perspective accounts of the academic disruption caused by ChatGPT across multiple universities and school districts. The facts regarding student usage, professor reactions, and the development of detection tools are clear and uncontradicted.
• Ambiguities identified: The exact number of students cheating globally is unknown, and the performance accuracy of detection tools like GPTZero and Turnitin's classifier is presented with some statistical uncertainty.
• Alternative interpretations: Some educators view the AI not as a cheating tool but as an educational aid similar to a calculator, suggesting the incident could be framed as a paradigm shift in teaching rather than purely a safety risk.

The widespread use of ChatGPT by students for academic cheating represents a significant disruption to educational institutions globally, forcing updates to academic integrity policies and teaching methods. However, the incident has negligible direct national security implications, posing no threat to critical infrastructure, intelligence, sovereignty, or economic security.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global
• Alleged perpetrator: Students

Threat characteristics

• Imminence: Long-term. This represents an ongoing educational and societal adjustment to new technology rather than an immediate national security crisis.
• Autonomy: Human-controlled. The AI tool acts as an assistant, generating text only when prompted and directed by individual student users.
• Novelty: Evolved capability. Academic plagiarism is an established issue, but the scale, coherence, and speed of generative AI represents a significant evolution in capability.

Impact by dimension

• Physical security: Negligible. The incident involves students using AI for academic cheating, which poses no threat to physical systems, critical infrastructure, or human safety.
• Information security: Negligible. No classified systems, intelligence collection methods, or state-sponsored information warfare operations were involved or compromised.
• Sovereignty: Negligible. The incident is confined to educational institutions and does not impact state authority, electoral systems, or core government decision-making.
• Economic security: Negligible. While affecting educational workflows, this academic plagiarism does not threaten national financial systems, strategic technology, or economic stability.
• Societal stability: Negligible. Academic dishonesty, while challenging for schools, does not cause mass surveillance, systematic discrimination, civil unrest, or large-scale threats to civil liberties.