Purportedly AI-Generated Images Reportedly Used During Four-Year Catfishing Campaign Targeting Welsh Teenager

A Welsh teenager reportedly endured a nearly four-year online impersonation campaign in which another teenager created fake social media accounts using her photographs and, later, AI-generated images depicting her. The campaign reportedly attracted more than 100,000 followers, led strangers to believe they knew the victim, and resulted in a High Court settlement and damages award.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The perpetrator used AI-generated imagery to impersonate the victim and manipulate others into believing they were in romantic relationships, causing targeted distress to the victim.

Additional risk subdomains

• 1.2 Exposure to toxic content: The victim was subjected to a sustained campaign of harassment and synthetic content generated to impersonate her, causing significant psychological distress.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The risk was caused by a human perpetrator who intentionally used AI-generated imagery to catfish and impersonate the victim after the AI model was deployed.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: Limited Risk. The report describes the use of AI-generated imagery to create deepfakes and impersonate a real person, which falls under the category of AI-generated content requiring transparency.

AI system and alleged parties

• AI system: None named
• AI purpose: Deepfake Image Generation; Image Generation
• Behaviour type: Tool
• Alleged developer: Synthetic media generation technology developers, Synthetic image generation technology developers, Deepfake technology developers
• Alleged deployer: Synthetic media creators, Online impersonators, Elha-Mai Weston, Deepfake creators, Catfishers
• Alleged harmed parties: Victims of deepfake abuse, Victims of catfishing, teenagers, Targets of online impersonation, Sasha Davies, Minors, Epistemic integrity

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Substantial, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report describes 1 occurrence resulting in a £10,000 compensation payout to the victim, though this is a legal settlement rather than a direct financial loss caused by the AI system.

Indirectly caused: Average financial loss per occurrence: £10,000 in legal settlement damages paid by the perpetrator to the victim.

Malicious content

Reported: The report explicitly describes toxic and malicious impersonation content created and spread directly by the perpetrator using AI.

Directly caused: The perpetrator created and spread AI-generated imagery of the victim to impersonate her across seven social media platforms, accumulating over 100,000 followers.

Civil rights

Reported: The report explicitly describes violations of human and civil rights, specifically harassment, misuse of private information, and breach of data protection legislation.

Directly caused: The victim's right to privacy and data protection was violated through the unauthorized use of her likeness and AI-generated imagery to impersonate her.

Privacy

Reported: The report explicitly describes privacy violations, including misuse of private information and breach of data protection legislation.

Directly caused: The perpetrator used photographs of the victim without her knowledge or consent to generate AI imagery and fake profiles, violating her personal privacy.

Psychological

Reported: The report explicitly describes significant psychological distress and suffering caused to the victim.

Directly caused: The victim suffered 'very significant distress and suffering' over four years, stopped feeling safe leaving her house, and felt her identity was stolen.

Indirectly caused: The men who were deceived into believing they were in romantic relationships with the fake persona likely suffered emotional distress upon discovering the truth.

Epistemic

Reported: The report explicitly describes epistemic harm through the creation of a fake online identity and synthetic evidence.

Directly caused: The perpetrator created a fictitious identity 'Sophie' using AI-generated imagery, misleading over 100,000 followers into believing the persona was real and that they had genuine connections with her.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 100000

Potential causes

Management

• Failure to Protect Users: Platform management failed to prioritize safety and identity protection.
• Inadequate Incident Response: Platforms only acted after public media exposure on the BBC.

Technology

• AI Image Generation: AI was used to generate realistic fake images of Sasha without her consent.
• Platform Identity Anonymity: Social media platforms allowed easy creation of anonymous fake profiles.

Data Inputs

• Unauthorized Photo Access: Sasha's public social media pictures were stolen to seed the fake profiles.
• Lack of Image Verification: Platforms did not verify if uploaded images belonged to the account creator.

Human Factors

• Malicious Impersonation Intent: The perpetrator intentionally targeted the victim to run a catfish campaign.
• User Deception: Followers believed the fake persona was real due to convincing AI images.

Process and Methods

• Inadequate Content Moderation: Platforms failed to remove fake profiles despite repeated reports by victim.
• Inadequate Police Investigation: Police treated online anonymity as an insurmountable barrier to action.

Regulatory Environment

• No Specific Catfishing Laws: The lack of specific catfishing laws hindered direct criminal prosecution.
• Weak Platform Regulation: Lack of regulatory pressure allowed platforms to ignore fake account reports.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear and consistent details about the catfishing campaign, the legal proceedings, the impact on the victim, and the specific role of AI-generated imagery. There is little ambiguity regarding the events or the parties involved.
• Ambiguities identified: The specific AI tool or generator used to create the imagery is not named.

A localized personal catfishing and harassment campaign where a UK teenager used AI-generated imagery to impersonate another teenager over four years. While causing significant personal distress and civil rights violations, the incident carries negligible national security implications and was resolved through standard civil litigation.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: Individual

Threat characteristics

• Imminence: Long-term. The case is resolved through a legal settlement, representing an ongoing societal capability concern rather than an active national security crisis.
• Autonomy: Human-controlled. The AI system acted purely as a tool for image generation, with the human perpetrator making all decisions regarding deployment and interaction.
• Novelty: Evolved capability. Catfishing is an established threat, but the integration of AI-generated synthetic imagery to maintain a multi-year impersonation represents an evolved capability.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or kinetic systems were targeted or affected in this incident.
• Information security: Negligible. The incident was a localized personal catfishing campaign and did not involve state-sponsored disinformation or intelligence compromises.
• Sovereignty: Negligible. No core government operations, electoral systems, or sovereign state authorities were impacted by this personal dispute.
• Economic security: Negligible. The incident resulted in a minor personal legal settlement and posed no threat to strategic industries or financial systems.
• Societal stability: Minor. While representing a severe violation of an individual's privacy and civil rights through harassment, the threat remains localized with minor national security implications.