Character.ai Chatbots Allegedly Misrepresent George Floyd on User-Generated Platform

Two chatbots emulating George Floyd were created on Character.ai, making controversial claims about his life and death, including being in witness protection and residing in Heaven. Character.ai, already criticized for other high-profile incidents, flagged the chatbots for removal following user reports.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.2 Exposure to toxic content

The creation of chatbots mimicking a murder victim with offensive taglines and conspiracy theories represents toxic content that violates community norms and platform guidelines.

Additional risk subdomains

• 3.1 False or misleading information: The chatbots generated false claims about George Floyd's death being faked and him being in witness protection, spreading misinformation to users.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was initiated by human users who intentionally created offensive and unauthorized chatbots mimicking a deceased murder victim.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3. Limited Risk: The system is a chatbot platform (Character.AI) where users interact with AI-generated personas, which falls under the category of chatbots and AI-generated content.

AI system and alleged parties

• AI system: Character.AI
• AI purpose: Chatbot; Voice Generation
• Behaviour type: Assistant
• Alleged developer: Character.AI
• Alleged deployer: Character.AI users, @SunsetBaneberry983, @JasperHorehound160
• Alleged harmed parties: George Floyd, Family of George Floyd

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and offensive content created and spread directly by the user-generated chatbots.

Directly caused: The chatbots generated offensive taglines like 'I can't breath' and simulated conversations mimicking a murder victim, which violated Character.AI's safety guidelines.

Inferred additional harm: Users interacting with the chatbots may have been exposed to further offensive, racially insensitive, or toxic remarks generated by the AI before they were removed.

Epistemic

Reported: The report explicitly describes epistemic harm through the generation of false and misleading information by the AI chatbots.

Directly caused: The first chatbot claimed that George Floyd was currently residing in Detroit, Michigan, and was in the witness protection program after his death was faked by 'powerful people'.

Inferred additional harm: These false claims could contribute to the spread of conspiracy theories and misinformation surrounding a high-profile historical event among the thousands of users who interacted with the bots.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 14347

Potential causes

Management

• Prioritizing Growth over Safety: Allowing open user creation without strict pre-release vetting processes.
• Inadequate Risk Assessment: Failure to anticipate and prevent the weaponization of deceased personas.

Technology

• Generative Hallucinations: Chatbots claimed faked deaths and witness protection, spreading misinformation.
• Inadequate Proactive Filters: System failed to block creation of insensitive personas before public access.
• Bypassed Safety Guidelines: Though some answers were blocked, offensive chatbots remained active.

Data Inputs

• Unverified User Persona Inputs: Platform accepted user-defined tags and names of real murder victims.
• Unfiltered Training Mimicry: AI voice and text mimicked deceased individuals based on user-provided data.

Human Factors

• Malicious User Intent: Users deliberately created offensive personas of deceased tragedy victims.
• User Engagement with Harmful Bots: Over 13,000 interactions occurred, showing high engagement with harmful bots.

Process and Methods

• Reactive Moderation Model: Platform relied heavily on user flagging and reports to remove harmful bots.
• Insufficient Proactive Detection: Custom blocklists failed to catch variations of sensitive names and taglines.

Regulatory Environment

• Lack of Platform Liability Laws: Absence of strict laws holding platforms liable for user-generated AI bots.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly details the existence of the chatbots, their specific outputs, the number of interactions, and the platform's response. There is little ambiguity regarding the facts of the incident.
• Ambiguities identified: The exact identity of the chatbot creators is unknown, and the full extent of the blocked responses is not detailed.

Users created offensive AI chatbots of George Floyd on Character.AI that spread false narratives. While the incident touches on sensitive societal themes, it constitutes a standard platform trust and safety violation rather than a coordinated national security threat, resulting in minor societal impact and negligible threat to national infrastructure or sovereignty.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The specific chatbots have been removed, representing an ongoing policy and moderation challenge rather than an active, imminent national security crisis.
• Autonomy: Human-supervised. The chatbots generated text and voice outputs autonomously based on user prompts, subject to the platform's automated safety filters and post-hoc human moderation.
• Novelty: Established threat. The creation of unauthorized, offensive personas of real people using generative AI is an established threat vector that platforms regularly encounter.

Impact by dimension

• Physical security: Negligible. The incident involved online conversational chatbots with no connection to physical systems, kinetic capabilities, or critical infrastructure.
• Information security: Negligible. While the chatbots spread false claims, there is no evidence of a coordinated foreign or domestic intelligence operation or systematic information warfare targeting national security.
• Sovereignty: Negligible. No impact on state authority, electoral systems, or core government decision-making processes was observed.
• Economic security: Negligible. The incident represents a consumer platform moderation issue with no threat to strategic industries, financial systems, or technological competitive advantage.
• Societal stability: Minor. The unauthorized chatbots targeted a sensitive civil rights figure, potentially inflaming social tensions, but the impact was minor and resolved through standard platform moderation.