AI-Generated Reading Summaries on Fable App Reportedly Wrote Biased and Offensive Commentary

Fable, a book-focused social app, used OpenAI’s API to generate AI-powered year-end reading summaries in December 2024. These summaries allegedly produced biased and offensive remarks about race, gender, and sexual orientation. Fable is reported to have apologized in a social media post on January 1, 2025.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The AI system generated outputs that unfairly stereotyped and discriminated against users based on protected characteristics like race, gender, and disability.

Additional risk subdomains

• 1.2 Exposure to toxic content: The AI exposed users to offensive, bigoted, and inappropriate language that violated community norms.
• 7.3 Lack of capability or robustness: The AI system failed to perform reliably, and its built-in safety filters failed to catch the offensive outputs.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The offensive outputs were generated post-deployment by the AI system itself, which was deployed by Fable to create playful summaries but unintentionally produced bigoted commentary.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The system is an AI-generated content feature (personalized summaries) which requires transparency obligations such as notifying users of AI interaction.

AI system and alleged parties

• AI system: OpenAI API (OpenAI)
• AI purpose: Automated Content Curation; Social Media Content Generation
• Behaviour type: Autonomous
• Alleged developer: OpenAI, Fable
• Alleged deployer: Fable
• Alleged harmed parties: Fable users, Fable

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Minor, indirect Negligible
• Differential treatment: direct Minor, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Malicious content

Reported: The report explicitly describes toxic and offensive content generated by the AI system.

Directly caused: The AI generated racist, sexist, and ableist comments, such as telling a Black reader to read white authors and mocking disability narratives.

Inferred additional harm: It is likely that many other offensive summaries were generated and displayed to users before the feature was taken down.

Differential treatment

Reported: The report explicitly describes differential treatment based on demographic reading choices.

Directly caused: The AI targeted users with derogatory or sarcastic remarks specifically because they read books by Black authors or authors with disabilities.

Inferred additional harm: Other minority or marginalized groups reading representative literature were likely subjected to similar targeted sarcasm or bigotry.

Psychological

Reported: The report explicitly describes psychological distress and offense caused to users who received the bigoted summaries.

Directly caused: Users like Tiana Trammell and Groves reported feeling stunned, flabbergasted, and disoriented by the offensive commentary targeting their race and reading choices.

Inferred additional harm: It is likely that other users among the 2 million base who received similar summaries experienced similar feelings of distress, alienation, and offense.

People affected

• Occurrences reported: 1
• People reportedly harmed: 3
• People reportedly exposed: 2000000

Potential causes

Management

• Prioritizing Automation Over Safety: Relying heavily on AI to scale personalized summaries without oversight.
• Underestimating Safety Work: Management underestimated the resources required for safe AI deployment.

Technology

• Failure of Offensive Filters: Filters for offensive language and topics failed to block the bad outputs.
• Inherent Model Biases: Underlying LLM carries societal biases that influenced generation.
• Unchecked Generative AI Output: AI directly pushed out content to users without automated validation.

Data Inputs

• Biased Training Data: Algorithms are built on data from a biased society, carrying those biases.
• User Reading History Input: Personalized reading lists triggered inappropriate demographic profiling.

Human Factors

• Programmer Biases: Developers' conscious or unconscious biases are carried into the machine.
• Underestimation of AI Risks: Fable team underestimated work needed for responsible AI deployment.

Process and Methods

• Lack of Human-in-the-Loop Review: Small team could not review 2 million individual summaries before release.
• Flawed Prompting Strategy: Prompting the AI to 'playfully roast' users led to offensive commentary.
• Inadequate Pre-release Testing: The feature lacked rigorous internal testing and safeguards before launch.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from multiple reputable sources (Wired, USA Today, NYT) provide consistent, detailed accounts of the incident, including direct quotes from affected users and Fable executives. The role of the AI system (OpenAI API) and the nature of the generated text are clearly documented, leaving very little ambiguity about what transpired.

The incident involves an AI-powered feature in a commercial book-tracking app that generated offensive and biased reading summaries. While causing distress to users and commercial disruption for the company, the event is entirely domestic and commercial in nature, presenting negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Unknown
• Primary target: No clear primary
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident has been resolved by the developer removing the feature, representing no ongoing or imminent national security crisis.
• Autonomy: Full autonomy. The AI system autonomously generated and delivered the personalized summaries directly to users without human-in-the-loop review.
• Novelty: Established threat. LLM generation of biased, offensive, or toxic content is a well-documented and established failure mode in generative AI deployments.

Impact by dimension

• Physical security: Negligible. No physical systems, infrastructure, or human safety threats were involved in this book-tracking app incident.
• Information security: Negligible. The incident involves localized, unintentional offensive content generation rather than state-sponsored information warfare or compromise of intelligence capabilities.
• Sovereignty: Negligible. No impact on state authority, electoral processes, or core government decision-making operations was reported.
• Economic security: Negligible. Minor commercial impact on a single book-tracking application with no systemic threat to financial systems, strategic industries, or national technological advantage.
• Societal stability: Negligible. While the AI generated biased and offensive commentary targeting protected characteristics, the impact was limited to users of a commercial app and does not threaten national-scale societal stability.