Education Scotland published curriculum reform evidence papers that contained approximately 20 fabricated references and significant statistical errors. The agency later confirmed that AI tools were used to edit these documents, though it remains unconfirmed whether the AI directly generated the fabrications. The incident has raised concerns regarding the reliability of government policy documents and the oversight of AI usage in public administration.
Education Scotland, the national education agency, published evidence papers for Scotland's school curriculum review. A University of Edinburgh lecturer identified about 20 references he believed did not exist. The agency corrected errors and disclosed that AI was used to edit the papers. Whether AI introduced any of the disputed references remains unconfirmed.
Risk classification
- Primary risk domain: 3 Misinformation
- Primary risk subdomain: 3.1 False or misleading information
The AI system inadvertently generated and spread incorrect information, including fabricated academic citations and false statistics, in official government policy papers.
Additional risk subdomains
- 7.3 Lack of capability or robustness: The AI system failed to perform reliably, leading to factual errors and hallucinations in a critical public administration application.
Causal factors
- Entity: AI
- Intent: Unintentional
- Timing: Post-deployment
The fabrication of references and statistical errors was an unexpected and unintentional outcome of using an AI system to edit official reports post-deployment.
EU AI Act risk tier
- Risk tier: 3 Limited Risk
Limited Risk: The system was used to generate and edit text content, which falls under AI-generated content requiring transparency about its AI origin.
AI system and alleged parties
- AI system: None named
- AI purpose: Writing Assistant; Automatic Copy-Editing
- Behaviour type: Assistant
- Alleged developer: Large language model developers
- Alleged deployer: Governments, Government of the United Kingdom, Government of Scotland, Government agencies, Education Scotland
- Alleged harmed parties: Scottish curriculum review stakeholders, Epistemic integrity, Educators in the United Kingdom, Educators in Scotland, Educators, Educational communities, Education policymakers
Harm severity
Severity ratings are based on this scale.
Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.
- Physical: direct Negligible, indirect Negligible
- Infrastructure: direct Negligible, indirect Negligible
- Property: direct Negligible, indirect Negligible
- Financial: direct Negligible, indirect Negligible
- Environmental: direct Negligible, indirect Negligible
- Malicious content: direct Negligible, indirect Negligible
- Differential treatment: direct Negligible, indirect Negligible
- Civil rights: direct Negligible, indirect Negligible
- Democracy: direct Negligible, indirect Negligible
- Privacy: direct Negligible, indirect Negligible
- Psychological: direct Negligible, indirect Negligible
- Epistemic: direct Minor, indirect Minor
- Child sexual exploitation and abuse: direct Negligible, indirect Negligible
Epistemic
Reported: The report explicitly describes epistemic harm, specifically the fabrication of academic references and incorrect statistics in official curriculum reports.
Directly caused: The AI-assisted reports contained approximately 20 fabricated references to non-existent publications and a false claim that over 200,000 pupils studied Higher PE.
Indirectly caused: The incident led to a public row and concerns over the reliability of the evidence base underpinning future educational reforms.
People affected
Potential causes
Management
- Lack of Operational Transparency: Management hid AI usage until external researchers flagged errors.
- Poor Risk Management: Leadership did not assess risks of using generative AI for evidence papers.
Technology
- AI Hallucination in Text Generation: The AI generated non-existent academic references and fabricated sources.
- Inaccurate Statistical Generation: AI produced impossible enrollment numbers contradicting actual exam data.
- Lack of Fact-Checking Mechanisms: No automated verification tools were integrated to check generated claims.
Data Inputs
- Unverified Training Data: AI models trained on unverified web data generated plausible-sounding lies.
- Lack of Grounded Reference Data: The AI system lacked access to real-time Scottish education databases.
Human Factors
- Overreliance on AI Outputs: Authors trusted AI-generated text without verifying the cited sources.
- Lack of AI Literacy: Staff lacked understanding of AI hallucination risks in policy drafting.
Process and Methods
- Inadequate Review of Drafts: Official policy documents were published without rigorous academic vetting.
- Delayed Response to Alerts: Officials brushed aside early warnings about errors for seven months.
Regulatory Environment
- Lack of AI Guidelines in Education: No clear regulatory frameworks governed AI use in public policy documents.
Information quality
- Classification confidence: High
- Reason for confidence: The report clearly details the nature of the AI's failure (fabricating references and statistics) and the context of its use (editing curriculum papers). While the specific AI model is not named, the impact and the reactions of the involved parties are well-documented.
- Ambiguities identified: It is not entirely clear whether the AI directly generated the fabricated references or if they were introduced during the editing process, though the report notes AI was used to edit them.
- Alternative interpretations: The fabrications could have been due to human error or plagiarism rather than AI hallucination, though Education Scotland confirmed AI was used.
Education Scotland published curriculum reform papers containing fabricated references and statistical errors due to unverified AI editing. This administrative failure represents a minor impact on government functions, highlighting the need for robust human oversight when integrating AI tools into public policy drafting, but carries negligible national security risk.
- Overall national security impact: Minor
- Scope: Single nation
- Primary target: United Kingdom
- Alleged perpetrator: Unknown
Threat characteristics
- Imminence: Long-term. Represents an ongoing strategic concern regarding quality assurance in government AI integration rather than an active national security crisis.
- Autonomy: Human-controlled. The AI system acted as an assistant to edit documents, with human administrators retaining ultimate control over publishing.
- Novelty: Established threat. AI hallucinations and the fabrication of academic references are well-documented and established behaviors of large language models.
Impact by dimension
- Physical security: Negligible. No physical systems, infrastructure, or human safety elements were affected by this administrative incident.
- Information security: Negligible. The incident involved unintentional AI hallucinations in local policy documents, not an adversarial information warfare campaign or intelligence compromise.
- Sovereignty: Minor. Minor impact on government functions as public administration documents contained errors, leading to political criticism and minor reputational damage to the education agency.
- Economic security: Negligible. No impact on strategic industries, financial systems, supply chains, or national technological competitive advantage.
- Societal stability: Negligible. No threats to civil liberties, mass surveillance implementation, or significant societal instability occurred.
.avif){kind=small}