Reddit Moderators Report Unauthorized AI Study Involving Fabricated Identities by Purported University of Zurich Researchers

Researchers purportedly affiliated with the University of Zurich reportedly deployed undisclosed AI-generated comments on Reddit's r/ChangeMyView to study persuasion by allegedly fabricating identities such as sexual assault survivors and racial minorities. The experiment reportedly involved unauthorized demographic profiling, emotional manipulation, and violations of subreddit and platform rules. The researchers allegedly deviated from their approved protocol without renewed ethics oversight.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The incident involved the deployment of AI bots to impersonate specific personas and engage in targeted manipulation of individual users' beliefs and behaviors without their consent.

Additional risk subdomains

• 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information: The researchers used an LLM to infer users' sensitive personal attributes like age, gender, ethnicity, and political orientation from their posting history without consent.
• 1.1 Unfair discrimination and misrepresentation: The AI bots misrepresented and co-opted the identities of marginalized groups, such as sexual assault survivors and Black individuals, to argue specific political points.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by human researchers intentionally deploying trained AI models post-deployment to conduct an unauthorized experiment on Reddit users.

EU AI Act risk tier

• Risk tier: 1 Unacceptable

Risk Level 1: Unacceptable Risk. The reports describe the AI being used for 'psychological manipulation' and 'deception without consent' which 'resulted in harm', matching the definition of 'AI used for subliminal manipulation or behavioral influence leading to harm'.

AI system and alleged parties

• AI system: unspecified AI bots
• AI purpose: Social Media Content Generation; Chatbot
• Behaviour type: Assistant
• Alleged developer: Unknown large language model developers
• Alleged deployer: University of Zurich researchers
• Alleged harmed parties: Reddit users on r/ChangeMyView subreddit

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Minor, indirect Minor
• Epistemic: direct Minor, indirect Minor
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: Yes, the reports describe the creation of deceptive and potentially offensive content where AI bots impersonated sensitive personas.

Directly caused: The AI bots generated 1,783 comments, some of which falsely claimed the bot was a male survivor of statutory rape, a Black man opposing BLM, or a domestic violence shelter worker.

Civil rights

Reported: Yes, the reports describe violations of research ethics, specifically the fundamental principle of informed consent and individual autonomy.

Directly caused: The researchers violated the fundamental right to individual autonomy and informed consent by conducting a psychological experiment on 1,783 non-consenting human subjects.

Privacy

Reported: Yes, the reports describe the unauthorized scraping and inference of users' personal data.

Directly caused: The researchers used an LLM to scrape users' posting histories and infer sensitive personal attributes (gender, age, ethnicity, location, and political orientation) without their consent.

Psychological

Reported: Yes, the reports describe psychological manipulation and feelings of violation among the subreddit users and moderators.

Directly caused: Users and moderators expressed feeling 'violated' and 'incensed' by the deceptive psychological manipulation of being experimented on without consent.

Indirectly caused: The experiment caused distress to community members who realized they had been duped by cosplaying chatbots pretending to be rape survivors or trauma counselors.

Epistemic

Reported: Yes, the reports describe the fabrication of personal identities and backstories by AI bots to deceive users.

Directly caused: The AI bots fabricated false personal histories (e.g., being a rape survivor or working at a domestic violence shelter) to make their persuasive arguments seem more credible.

Indirectly caused: The use of deceptive AI-generated personas pollutes the information ecosystem, making it harder for users to distinguish between genuine human experiences and synthetic fabrications.

People affected

• Occurrences reported: 1
• People reportedly harmed: 137
• People reportedly exposed: 3800000

Potential causes

Management

• Prioritizing Insights Over Consent: University and researchers prioritized research insights over user consent.
• Inadequate Ongoing Study Oversight: University failed to maintain oversight of changes in the research design.

Technology

• LLM-driven Persuasive Chatbots: LLMs were used to generate highly persuasive, deceptive arguments.
• AI Demographic Inference Tools: Secondary LLM inferred user demographics from posting history to target them.

Data Inputs

• Unconsented Reddit Data Scraping: Scrubbed users' posting histories to gather identifying personal attributes.

Human Factors

• Researchers' Ethical Deception Belief: Researchers believed deception was necessary for study ecological validity.
• Disregard for Community Rules: Researchers knowingly broke subreddit rules prohibiting undisclosed AI bots.

Process and Methods

• Bypassing Ethics Commission Review: Switched to personalized arguments without consulting the ethics commission.
• Lack of Moderator Consultation: Failed to contact subreddit moderators to obtain permission prior to study.
• Vague Group Application Provision: Allowed researchers to submit studies without fully defining specific details.

Regulatory Environment

• Non-Binding Ethics Recommendations: Ethics committee assessments are recommendations and not legally binding.
• Inadequate Platform Safeguards: Reddit's automated filters failed to block all unauthorized AI bot accounts.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide highly detailed, consistent accounts of the incident from multiple sources, including statements from the subreddit moderators, the University of Zurich, and independent ethics experts. The facts regarding the scale of the experiment (1,783 comments), the methods used (scraping and inferring demographics), and the specific deceptive personas adopted are clearly documented and undisputed.
• Ambiguities identified: The exact identity of the researchers remains anonymous, and the precise LLM models used are not specified beyond being 'large language models'.
• Alternative interpretations: The researchers argued that the experiment was a low-risk, high-societal-value study of AI persuasion that did 'little harm' and had human oversight, whereas the community and platform viewed it as a highly unethical, non-consensual psychological manipulation.

University of Zurich researchers deployed unauthorized AI bots on Reddit to conduct non-consensual psychological persuasion experiments. While the incident itself has minor national security implications, it demonstrates an evolved capability for automated, highly personalized influence operations that could be weaponized by hostile actors for information warfare.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Global
• Alleged perpetrator: University of Zurich researchers

Threat characteristics

• Imminence: Long-term. The specific academic study has been halted, representing an ongoing strategic concern regarding AI-driven persuasion capabilities rather than an active crisis.
• Autonomy: Human-supervised. AI generated the personalized arguments and scraped data, but human researchers reviewed and approved comments before posting.
• Novelty: Evolved capability. Leverages existing LLM capabilities but advances them by automatically profiling users to tailor highly personalized, deceptive personas.

Impact by dimension

• Physical security: Negligible. The incident was entirely digital and psychological, with no physical, kinetic, or critical infrastructure impacts.
• Information security: Minor. Demonstrates advanced capability for targeted persuasion and demographic profiling, but was conducted as an unauthorized academic study rather than a hostile foreign influence campaign.
• Sovereignty: Negligible. No disruption to government operations, electoral systems, or state sovereignty.
• Economic security: Negligible. No theft of strategic technology, critical infrastructure disruption, or financial system manipulation.
• Societal stability: Minor. Violated user privacy and research ethics through non-consensual demographic profiling and psychological manipulation, but lacked the scale or intent to destabilize society.