South Africa Draft National AI Policy Reportedly Included Fictitious References Believed to Be AI Hallucinations

South Africa's Draft National AI Policy, gazetted for public comment, reportedly contained at least six fictitious academic references. Several cited articles or journals reportedly did not exist or were disclaimed by journal editors, and experts said the errors were consistent with AI hallucinations. The Department of Communications and Digital Technologies said it was reviewing the discrepancies but argued they did not affect the draft's substance.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The AI system generated completely fictitious academic citations that were included in a draft national policy document, spreading false information.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Government officials relied on the AI tool to generate references for a national policy document without verifying the outputs, demonstrating unsafe use and overreliance.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by an AI tool hallucinating fictitious academic citations during the drafting process, which was an unintentional outcome of using the model.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a general-purpose large language model used to generate text and citations, which falls under AI-generated content requiring transparency and verification to prevent misleading the public.

AI system and alleged parties

• AI system: AI tool
• AI purpose: Writing Assistant; Technical Text Generation
• Behaviour type: Assistant
• Alleged developer: Unknown large language model developers, Unknown generative AI developers
• Alleged deployer: Government of South Africa, Department of Communications and Digital Technologies (South Africa)
• Alleged harmed parties: General public of South Africa, General public, Epistemic integrity, AI policy stakeholders

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: The report explicitly describes epistemic harm in the form of AI-generated hallucinations and fabricated academic citations published in an official government document.

Directly caused: The AI tool generated at least six completely fictitious academic citations, fabricating titles, authors, and journal volumes that do not exist.

Indirectly caused: The publication of these fabricated citations in the Draft National AI Policy misled readers and required academic editors and researchers to spend time verifying and debunking the false references.

Inferred additional harm: It is likely that other references in the 67-citation list are also fabricated, potentially spreading further false academic records if not fully audited.

People affected

• Occurrences reported: 1
• People reportedly exposed: 6

Potential causes

Management

• Lack of Quality Oversight: Management failed to implement rigorous verification steps before publication.
• Downplaying of Errors: Management minimized the significance of fabricated scientific references.

Technology

• AI Model Hallucination: AI models predict plausible-sounding but fictitious academic citations.

Data Inputs

• Unverified Training Data: AI model predicts references based on patterns rather than factual databases.

Human Factors

• Lack of Human Supervision: Users relied on AI outputs without verifying the existence of cited sources.
• Irresponsible AI Use: Drafting policy documents using AI tools without proper care or disclosure.

Process and Methods

• Inadequate Citation Review: The draft policy review process failed to verify the academic references.

Regulatory Environment

• Absence of AI Use Standards: No clear regulations governing the use of AI in government policy drafting.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, verified evidence from multiple academic journal editors confirming that the citations in the draft policy are indeed fabricated. The department's response and the analysis by AI experts strongly support the conclusion of AI hallucination due to lack of human oversight.
• Ambiguities identified: The specific AI model used by the department is not identified, and the exact mechanism of how the citations were generated (whether the entire text or just the references were AI-generated) remains unconfirmed by the department.
• Alternative interpretations: It is theoretically possible, though highly unlikely, that the references were fabricated manually by a human writer, but the pattern of generating plausible-sounding but non-existent journals and volumes is highly characteristic of LLM hallucinations.

The South African government's Draft National AI Policy was found to contain fabricated academic citations due to AI hallucinations. This incident represents a minor, self-inflicted blow to government credibility and highlights the risks of overreliance on AI tools in public administration without rigorous human verification, but poses no significant threat to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: South Africa
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents an ongoing strategic concern regarding the integration of AI tools in public administration rather than an immediate crisis.
• Autonomy: Human-supervised. The AI system acted as a writing assistant to generate text, which was intended to have human oversight and verification before publication.
• Novelty: Established threat. AI model hallucination and the generation of plausible-sounding but false information is a well-documented and established characteristic of LLMs.

Impact by dimension

• Physical security: Negligible. No threats to physical systems, infrastructure, or human safety were reported in connection with this incident.
• Information security: Negligible. No evidence of foreign intelligence compromise, classified data theft, or coordinated disinformation campaigns.
• Sovereignty: Minor. Minor impact on government policy-making credibility due to the inclusion of fabricated citations in a draft national policy document, manageable through standard review processes.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or direct economic stability threats resulted from this incident.
• Societal stability: Negligible. No indicators of mass surveillance, civil unrest, or systemic human rights violations are present in this case.