Sydney High Schooler Allegedly Generated Deepfakes of Other Students

A Sydney high school student allegedly used AI platforms to create explicit deepfake images of female classmates, which were then distributed through fake social media accounts. The incident reportedly caused significant distress among the victims, leading to an investigation by Campbelltown city police area command and involvement from the eSafety Commissioner.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The student used AI to generate explicit deepfakes of classmates for targeted humiliation and harassment, which falls under targeted manipulation and creation of humiliating or sexual imagery.

Additional risk subdomains

• 1.2 Exposure to toxic content: The generated explicit deepfakes constitute toxic and sexually explicit content distributed online.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by a human student intentionally using deployed AI platforms to generate and distribute explicit deepfake images of classmates.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The incident involves 'AI-generated content such as deepfakes', which falls under Risk Level 3 (Limited Risk) requiring transparency obligations.

AI system and alleged parties

• AI system: artificial intelligence platforms
• AI purpose: Deepfake Image Generation; Image Generation
• Behaviour type: Tool
• Alleged developer: Unknown deepfake technology developers
• Alleged deployer: Unknown Sydney area high school student
• Alleged harmed parties: Anonymous Sydney area high school students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Substantial
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Minor
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Substantial, indirect Negligible

Malicious content

Reported: Yes, the report explicitly describes toxic or malicious content created and spread directly by the incident.

Directly caused: A student used AI platforms to create explicit deepfake images of female classmates.

Indirectly caused: The explicit deepfakes were circulated online using fake social media accounts, exposing others to toxic content.

Inferred additional harm: It is likely that dozens of students and social media users were exposed to this non-consensual explicit content before it was investigated.

Privacy

Reported: Yes, the report explicitly describes privacy violations caused indirectly by the incident.

Indirectly caused: The distribution of explicit deepfakes violated the personal privacy and bodily autonomy of the targeted female students.

Inferred additional harm: The creation of deepfakes using the likeness of the female students constitutes a severe, non-consensual violation of their privacy.

Psychological

Reported: Yes, the report explicitly describes psychological or mental-health harm to people caused indirectly by the incident, noting that the school is helping affected students with appropriate wellbeing support.

Indirectly caused: The distribution of explicit deepfakes of female students caused distress and psychological harm, requiring the school to provide wellbeing support.

Inferred additional harm: It is highly likely that the affected female students experienced significant anxiety, humiliation, and trauma due to the non-consensual sexual content.

Child sexual exploitation and abuse

Reported: Yes, the report explicitly describes a CSEA incident where a student used AI to create pornography featuring other students, who are minors.

Directly caused: A teenage student used AI to generate explicit deepfake images of other minor students.

Inferred additional harm: Since the victims are high school students, the generated explicit images constitute child sexual abuse material (CSAM).

People affected

• Occurrences reported: 1
• People reportedly harmed: 5
• People reportedly exposed: 50

Potential causes

Technology

• AI Deepfake Platforms: AI platforms were used to generate explicit pornography of students.

Data Inputs

• Target Student Images: Photos of female students were used as inputs for the AI tool.

Human Factors

• Student Misconduct: The student chose to create and distribute harmful non-consensual images.

Process and Methods

• Fake Social Media Distribution: Fake social media accounts were used to circulate the deepfake images.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the nature of the incident, the actors involved, and the harm caused. While the specific AI platform is not named, the classification of the risk and behavior is straightforward based on the detailed description of deepfake creation and distribution.
• Ambiguities identified: The specific AI platforms used and the exact number of affected students are not specified.

A localized incident involving a high school student in Australia using generative AI to create non-consensual explicit images of classmates. While presenting serious local law enforcement, privacy, and child protection concerns, the incident has negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. This represents an ongoing strategic concern regarding the accessibility of generative AI tools rather than an active national security crisis.
• Autonomy: Human-controlled. The AI systems were operated directly as tools by an individual user to generate specific content.
• Novelty: Established threat. The misuse of generative AI for creating non-consensual explicit imagery of individuals is a known and recurring issue.

Impact by dimension

• Physical security: Negligible. The incident does not involve kinetic attacks, autonomous weapons, critical infrastructure manipulation, or any threats to physical systems.
• Information security: Negligible. There is no indication of state-sponsored disinformation, compromise of intelligence capabilities, or theft of classified models.
• Sovereignty: Negligible. The event is a localized criminal matter and does not threaten state authority, borders, electoral systems, or core government operations.
• Economic security: Negligible. The incident has no impact on strategic industries, macro-financial stability, or national technological competitiveness.
• Societal stability: Minor. While the creation of explicit deepfakes of minors represents a serious violation of individual privacy and human rights, the impact is localized and manageable within standard law enforcement and educational frameworks.