ChatGPT Reportedly Introduces Errors in Critical Child Protection Court Report

A child protection worker in Victoria, Australia reportedly used ChatGPT to draft a report submitted to the Children's Court. The purportedly AI-generated report contained inaccuracies and downplayed risks to the child, allegedly resulting in a privacy breach when sensitive information was shared with OpenAI.

Risk classification

• Primary risk domain: 5 Human-Computer Interaction
• Primary risk subdomain: 5.1 Overreliance and unsafe use

The worker inappropriately relied on a general-purpose chatbot to draft a highly sensitive child protection risk assessment in a critical legal context, leading to unsafe and inaccurate outputs.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The worker disclosed highly sensitive personal information of a child, parents, and carer to OpenAI by inputting it into ChatGPT.
• 7.3 Lack of capability or robustness: ChatGPT failed to understand the context of sexual abuse and hallucinated that a doll used for sexual purposes was an indication of positive parenting.

Causal factors

• Entity: Human
• Intent: Unintentional
• Timing: Post-deployment

The incident was initiated by a human worker's decision to use an unauthorized tool and input sensitive data, with the resulting privacy breach and inaccurate report being unintentional outcomes of trying to save time.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: ChatGPT is a chatbot, which falls under Risk Level 3 requiring specific transparency obligations, though its unsafe use in a child protection context highlights high-risk implications.

AI system and alleged parties

• AI system: ChatGPT (OpenAI)
• AI purpose: Writing Assistant; Child Welfare Risk Assessment
• Behaviour type: Assistant
• Alleged developer: OpenAI
• Alleged deployer: Government of Victoria, Employee of Department of Families Fairness and Housing, Department of Families Fairness and Housing
• Alleged harmed parties: Unnamed family of child, Unnamed child

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Privacy

Reported: The report explicitly describes privacy violations caused directly by the incident.

Directly caused: The case worker entered highly sensitive personal information, including names and risk assessments of a child, mother, father, and carer, into ChatGPT, disclosing it to OpenAI.

Inferred additional harm: If ChatGPT was used in the other 100 suspected cases within the unit, hundreds of other individuals' private data may have been disclosed.

People affected

• Occurrences reported: 1
• People reportedly harmed: 4
• People reportedly exposed: 4

Potential causes

Management

• Lack of Employee Training: Management failed to provide training on GenAI privacy and accuracy risks.
• Insufficient Risk Assessment: Management did not assess risks of GenAI across different functions.
• No Visibility into AI Usage: DFFH had no way to monitor what data employees entered into GenAI tools.

Technology

• Probabilistic Nature of LLMs: LLMs prioritize sequence probability over factual accuracy and context.
• Inability to Understand Context: ChatGPT failed to grasp the context of sexual abuse, misinterpreting a risk.
• Inaccurate Content Generation: The AI hallucinated positive parenting strengths from details of abuse.

Data Inputs

• Input of Sensitive Client Data: Worker entered names and risk assessments into a public AI tool.
• Data Shared with Third Party: Sensitive data was disclosed to OpenAI, releasing it from DFFH control.

Human Factors

• Desire to Save Time: Worker used ChatGPT to save time and present work more professionally.
• Lack of AI Literacy: Staff did not understand risks of entering sensitive data into public AI.
• Over-reliance on AI Output: Worker trusted and submitted AI-generated text containing critical errors.

Process and Methods

• Lack of Technical Access Controls: DFFH had no IP or DNS blocking to prevent access to public GenAI tools.
• Inadequate Review of Reports: Draft with major AI errors was submitted to court before legal review.
• No Dedicated GenAI Policies: DFFH had no specific rules regarding when and how GenAI should be used.

Regulatory Environment

• Inadequate External AI Regulations: Lack of strict statutory rules governing public sector use of GenAI.
• Privacy Principle Contravention: DFFH failed to meet IPP 3.1 and 4.1 standards for data protection.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly detailed and originate from the official investigation by the Office of the Victorian Information Commissioner (OVIC), providing clear facts, specific quotes, and regulatory findings.
• Ambiguities identified: The worker denied entering personal information, but OVIC concluded on the balance of probabilities that they did based on co-worker testimonies.
• Alternative interpretations: None. The facts of the unauthorized use and the specific hallucination are well-documented and accepted by the department.

A child protection worker in Australia used ChatGPT to draft a sensitive court report, leaking private data and generating severe hallucinations that downplayed child abuse. The incident highlights established risks of shadow AI use in government processes, resulting in domestic regulatory action but presenting negligible direct threat to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. The immediate crisis is resolved with a regulatory ban, representing a long-term policy and governance challenge regarding shadow AI use.
• Autonomy: Human-supervised. The AI system autonomously generated the report text, which was then subject to human review and oversight before court submission.
• Novelty: Established threat. Data leakage and AI hallucinations in professional drafting are well-documented and established risks of generative AI technologies.

Impact by dimension

• Physical security: Negligible. No physical threat, kinetic attacks, or critical infrastructure compromise occurred in this incident.
• Information security: Negligible. The incident involved domestic privacy breaches of sensitive personal data, but did not compromise national intelligence capabilities or classified military systems.
• Sovereignty: Minor. A local government agency's court drafting process was compromised by AI inaccuracies, leading to a regulatory ban, but it was resolved internally without disrupting core national sovereign functions.
• Economic security: Negligible. The incident did not involve strategic technology theft, financial system disruption, or threats to national economic security.
• Societal stability: Minor. Violated individual privacy rights and risked child welfare due to inaccurate reports, but impact was localized to specific cases and did not threaten population-scale societal stability.