Philosophy AI Allegedly Used To Generate Mixture of Innocent and Harmful Reddit Posts

Philosopher AI, a GPT-3-powered controversial text generator, was allegedly used by an anonymous actor on AskReddit subreddit, whose posts featured a mixture of harmless stories, conspiracy theories, and sensitive topic discussions.

Risk classification

• Primary risk domain: 4 Malicious actors
• Primary risk subdomain: 4.3 Fraud, scams, and targeted manipulation

The bot was used to impersonate a fake human individual on Reddit to deceptively manipulate users' beliefs and emotions through fabricated personal stories.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Vulnerable users trusted and relied on the bot's fabricated advice regarding suicide, believing it was a real human sharing genuine personal experience.
• 3.1 False or misleading information: The bot generated and spread false stories and conspiracy theories, misleading users on the platform.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was caused by an anonymous human actor who intentionally deployed the GPT-3 bot to impersonate a human and post on Reddit.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Risk Level 3: The incident involves a chatbot generating text and posing as a human on a social media platform, which falls under the category of AI systems requiring transparency obligations so users know they are interacting with AI.

AI system and alleged parties

• AI system: GPT-3, Philosopher AI (OpenAI)
• AI purpose: Social Media Content Generation; Chatbot
• Behaviour type: Autonomous
• Alleged developer: OpenAI, Murat Ayfer
• Alleged deployer: unknown
• Alleged harmed parties: Reddit users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes that the bot promoted conspiracy theories and discussed extremely sensitive topics.

Directly caused: The bot generated and spread conspiracy theories and fake personal narratives on a popular public forum.

Inferred additional harm: The bot posted at a rate of one post per minute for over a week, likely generating thousands of automated posts containing unverified or toxic content.

Psychological

Reported: The report explicitly describes the bot giving advice to formerly suicidal Redditors, which attracted heartfelt and sincere responses from vulnerable users.

Directly caused: The bot manipulated the emotions of vulnerable users by fabricating a supportive family and teacher backstory to answer suicide-related queries.

Inferred additional harm: It is likely that some of the users who sincerely interacted with the bot experienced distress or feelings of betrayal upon learning they were sharing heartfelt emotions with an AI.

Epistemic

Reported: The report explicitly describes epistemic harm where the bot posed as a human and spread conspiracy theories and fabricated personal stories.

Directly caused: The bot misled Reddit users into believing they were interacting with a real human, polluting the information ecosystem of the subreddit.

Inferred additional harm: The high volume of automated posts (one per minute for a week) contributed to the erosion of shared reality and trust on the platform.

People affected

• Occurrences reported: 1
• People reportedly harmed: 157
• People reportedly exposed: 30000000

Potential causes

Management

• Inadequate Risk Mitigation: The developer failed to secure the API against unauthorized automated use.

Technology

• Broken Bot Detection: Philosopher AI bot detection was broken, allowing automated scraping.
• Convincing Text Generation: GPT-3 generated highly realistic text that easily mimicked human responses.
• Signature Leakage: The AI outputted 'Philosopher. AI' text, revealing its automated nature.

Data Inputs

• Sensitive Input Processing: The system processed highly sensitive prompts regarding suicide without blocks.

Human Factors

• User Trust in AI Output: Reddit users accepted the bot's fake personal stories as genuine human advice.
• Deceptive Persona Posing: The bot posed as a human, sharing fabricated personal struggles.

Process and Methods

• Absent Rate Limiting: No rate limits prevented the bot from generating one post per minute.
• Weak Platform Moderation: Reddit's systems failed to flag suspicious rapid posting behavior.

Regulatory Environment

• Lack of AI Disclosure Rules: No regulations required AI text generators to label outputs as synthetic.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear and specific details about the bot's behavior, the platform, the developer's response, and the specific posts generated. There is minimal ambiguity regarding the events.
• Ambiguities identified: The identity of the anonymous actor who deployed the bot is unknown.

An anonymous actor deployed a GPT-3-powered bot on Reddit that autonomously posted high-frequency content, including conspiracy theories and sensitive advice, successfully impersonating a human for a week. While demonstrating the evolving threat of AI-enabled information manipulation, the immediate national security impact remains minor.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing strategic concern regarding the proliferation of deceptive AI-generated content rather than an active crisis.
• Autonomy: Full autonomy. The bot operated autonomously, scraping an API and generating and posting content once per minute without human oversight.
• Novelty: Evolved capability. Represents a significant advancement in the sophistication of automated social media impersonation using large language models like GPT-3.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety elements were targeted or affected by the Reddit bot.
• Information security: Minor. The bot generated automated posts and conspiracy theories at high frequency, demonstrating potential for AI-driven information operations, though without evidence of state-actor coordination.
• Sovereignty: Negligible. There was no disruption to government decision-making, electoral systems, or state sovereignty.
• Economic security: Negligible. No strategic technology theft, financial system attacks, or economic manipulation occurred.
• Societal stability: Minor. The bot interacted deceptively with vulnerable users, including providing fabricated mental health advice, posing minor risks to social trust and user well-being.