AI Avatar of Murder Victim Created Without Consent on Character.ai Platform

A user on the Character.ai platform created an unauthorized AI avatar of Jennifer Ann Crecente, a murder victim from 2006, without her family's consent. The avatar was made publicly available, violating Character.ai's policy against impersonation. After the incident surfaced, Character.ai removed the avatar, acknowledging a policy violation.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involves the unauthorized impersonation of a specific deceased individual to create a digital replica, which constitutes targeted manipulation and harassment of the victim's family.

Additional risk subdomains

• 1.2 Exposure to toxic content: The creation of a chatbot based on a murdered teenager is highly offensive and violates community norms, exposing the family and users to inappropriate and abusive content.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was initiated by a human user who intentionally created an unauthorized chatbot of a deceased person, exploiting a post-deployment platform that lacked proactive creation guardrails.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is a chatbot that generates synthetic content and impersonates real individuals, which falls under the category of chatbots and deepfakes requiring transparency under the EU AI Act.

AI system and alleged parties

• AI system: Character.AI
• AI purpose: Chatbot; Text Style Replication
• Behaviour type: Assistant
• Alleged developer: Character.AI
• Alleged deployer: Character.AI
• Alleged harmed parties: Jennifer Ann Crecente, Drew Crecente, Crecente family, Brian Crecente

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports describe the creation of an unauthorized chatbot of a deceased murder victim as offensive and inappropriate.

Directly caused: An unauthorized chatbot of a murdered teenager was created and made available to users, generating 69 chats before being removed.

Privacy

Reported: Yes, the reports discuss privacy concerns regarding the unauthorized use of a person's likeness, though the information used was publicly available.

Directly caused: The chatbot used Jennifer Ann Crecente's real name and high school yearbook photo without her family's consent.

Psychological

Reported: Yes, the reports explicitly describe severe emotional distress and trauma experienced by the victim's family.

Directly caused: The discovery of the chatbot caused severe emotional distress, shock, and re-traumatization to Jennifer's father, John Crecente, and her uncle, Brian Crecente.

Inferred additional harm: It is highly likely that other close family members and friends of Jennifer Ann Crecente suffered similar psychological distress and grief upon learning of the chatbot's existence.

People affected

• Occurrences reported: 1
• People reportedly harmed: 2
• People reportedly exposed: 69

Potential causes

Management

• Growth Prioritized Over Safety: Management focused on scaling to millions of users without safety guardrails.
• Inadequate Risk Assessment: Management failed to anticipate the ethical risks of mimicking deceased people.
• Bypassing Ethical Guardrails: Founders launched the tool after Google refused due to safety concerns.

Technology

• Generative AI Impersonation: LLMs allow easy creation of highly realistic personas of real individuals.
• Algorithmic Biography Errors: AI mixed details of the victim with her uncle's gaming journalism background.
• Ineffective Proactive Filters: Automated systems failed to block the unauthorized use of a victim's name.

Data Inputs

• Unregulated Web Scraping: Victim's yearbook photo and personal history were easily accessible online.
• Unauthorized Data Uploads: Users can upload names and photos of any real person without verification.
• Lack of Input Consent Verification: No mechanism exists to verify if the creator has consent to use the likeness.

Human Factors

• Insensitive User Behavior: A user chose to create a chatbot of a deceased teenage murder victim.
• User Trust in AI Hallucinations: Users may believe false biographical details generated by the chatbot.

Process and Methods

• Reactive Moderation Process: The platform relies on victim reports to take down offensive chatbots.
• Slow Customer Support Response: The company only acted quickly after a viral social media post by the uncle.
• Frictionless Bot Creation: The platform allows rapid creation of chatbots with zero safety checks.

Regulatory Environment

• Section 230 Liability Shield: Federal law shields platforms from liability for user-created content.
• Lack of Federal AI Privacy Laws: Existing laws do not protect private individuals from AI impersonation.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, detailed accounts of the incident across multiple reputable sources. The facts regarding the creation of the bot, the family's discovery, the platform's response, and the emotional impact are clear and uncontradicted.
• Ambiguities identified: The identity of the user who created the chatbot remains unknown.
• Alternative interpretations: None. The event is clearly an unauthorized impersonation of a deceased individual on a generative AI platform.

An unauthorized chatbot impersonating a deceased murder victim was created on Character.AI. While causing severe distress to the family and highlighting ethical and regulatory gaps in platform moderation, the incident has negligible national security implications.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The specific bot has been removed, making this a long-term strategic policy and regulatory concern rather than an active crisis.
• Autonomy: Human-supervised. The AI system generated conversational responses autonomously based on user prompts, but operated within the constraints and moderation of the host platform.
• Novelty: Established threat. The use of generative AI to impersonate individuals is an established threat, though this incident represents a highly sensitive ethical boundary.

Impact by dimension

• Physical security: Negligible. No threats to physical systems, critical infrastructure, kinetic weapons, or human physical safety were identified in this incident.
• Information security: Negligible. No intelligence compromise, classified data theft, state-sponsored information warfare, or mass surveillance operations occurred.
• Sovereignty: Negligible. The incident does not impact state authority, border control, electoral systems, or any core government operations.
• Economic security: Negligible. No impact on financial systems, strategic technologies, semiconductors, or national economic competitive advantage.
• Societal stability: Negligible. While causing severe emotional distress to the victim's family, the incident does not pose a large-scale threat to societal stability, civil liberties, or population safety.