Chatbot in Workplace Training at Bunbury Prison Reveals Real Names in Sexual Harassment Case

During workplace training at Bunbury Prison in Western Australia, a trainer used Microsoft's Copilot AI chatbot to generate case study scenarios. The chatbot produced a scenario that included the real name of a former employee involved in a sexual harassment case, revealing sensitive information.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system failed to perform reliably by failing to adhere to the negative constraint of generating a fictional scenario, instead regurgitating real-world sensitive data and falsely labeling it as fictional.

Additional risk subdomains

• 3.1 False or misleading information: The AI system explicitly generated false meta-information by claiming 'this case study is entirely fictional' when it actually contained real names and active court details, misleading the user.
• 1.2 Exposure to toxic content: The AI system exposed users in a workplace training environment to sensitive, triggering, and inappropriate real-world sexual harassment details involving their former colleague.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by Microsoft's Copilot chatbot generating real-world names and details while claiming the scenario was fictional, which was an unexpected and unintentional outcome of the generation process.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3. Limited Risk: The system involved is a chatbot (Microsoft Copilot) generating text content. According to the definitions, chatbots and AI-generated content systems pose a moderate risk and are subject to transparency obligations to ensure users are aware they are interacting with AI.

AI system and alleged parties

• AI system: Copilot (Microsoft)
• AI purpose: Chatbot; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: Microsoft
• Alleged deployer: Western Australia Department of Justice, Charlotte Ingham
• Alleged harmed parties: Western Australia Department of Justice senior staff members, Western Australia Department of Justice, Bronwyn Hendry

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes the creation and spread of sensitive and triggering content.

Directly caused: The AI system generated a case study detailing real-world sexual harassment and bullying allegations using real names, which was then presented to prison staff.

Psychological

Reported: The report explicitly describes psychological harm to the victim of the sexual harassment case.

Directly caused: The victim ([PERSON_004]) experienced direct psychological distress, describing the presentation of her real-world trauma in her former workplace as 'very triggering'.

Indirectly caused: The trainer experienced professional embarrassment and distress upon realizing she had presented a real-world sensitive scenario as fictional.

Epistemic

Reported: The report explicitly describes epistemic harm in the form of fabrication and misleading labeling.

Directly caused: The AI system generated a real-world scenario while falsely asserting that 'this case study is entirely fictional', directly misleading the trainer into presenting it.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 15

Potential causes

Management

• Inadequate Vendor Oversight: Department failed to audit training materials provided by the contractor.
• Lack of AI Policy: Organization did not establish rules for using generative AI in training.

Technology

• Regurgitation of Training Data: Chatbot outputted real names and case details instead of fictional ones.
• Misleading System Assurances: AI falsely claimed the generated case study was 'entirely fictional'.
• Lack of Output Verification: No built-in safety filter to prevent leaking real-world litigation names.

Data Inputs

• Ingestion of Active Court Data: System trained on public search data containing sensitive court cases.
• Specific Prompting Vulnerability: Detailed prompts caused chatbot to retrieve and insert real people's names.

Human Factors

• Overreliance on AI Output: Trainer assumed the generated scenario was fictional without verifying.
• Lack of AI Literacy: Trainer did not know generative AI could hallucinate or regurgitate data.

Process and Methods

• Inadequate Content Review: No validation of training materials before presenting them to employees.
• Lack of AI Guidelines: Department lacked policies on prompting and checking AI outputs.

Regulatory Environment

• Absence of AI Safety Standards: No clear regulatory frameworks preventing regurgitation of sensitive data.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly details the AI system used (Microsoft Copilot), the developer (Microsoft), the context of the incident (workplace training at Bunbury prison), the specific output failure (regurgitating real names/details while claiming to be fictional), and the direct psychological impact on the victim. The ABC's independent replication provides strong corroboration of the system's behavior.
• Ambiguities identified: None significant, though the exact number of attendees in the training session is not specified.
• Alternative interpretations: One could argue this is a privacy incident, but since the information was publicly available from court proceedings, it is classified under capability failure and epistemic/toxic content rather than a privacy breach per the boundary rules.

A Microsoft Copilot chatbot regurgitated the real name and details of a harassment victim during a prison staff training session in Australia. While causing localized distress and highlighting LLM data privacy risks, the incident poses negligible threats to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding LLM data privacy and regurgitation rather than an active crisis.
• Autonomy: Human-controlled. The AI acted as a writing assistant to generate content prompted by a human user, who retained control over whether to present the slides.
• Novelty: Established threat. Data regurgitation, hallucinations, and lack of constraint adherence are well-documented behaviors in large language models.

Impact by dimension

• Physical security: Negligible. No threat to physical systems, critical infrastructure, or kinetic safety occurred during this workplace training incident.
• Information security: Negligible. No espionage, intelligence compromise, or systematic information warfare campaign was identified in this incident.
• Sovereignty: Negligible. The incident occurred within a state government department training session but did not disrupt core sovereign functions, electoral systems, or state decision-making.
• Economic security: Negligible. No strategic technology theft, financial system disruption, or threat to national economic stability was associated with this localized tool failure.
• Societal stability: Minor. Minor localized impact on privacy and psychological safety, as the AI regurgitated sensitive personal data of a harassment victim in a government workplace training context.