Fin AI Support Chatbot Deployed by Anthropic Reportedly Prolonged a Canadian Engineering Firm's $3,500 Billing Dispute

An executive at a Canadian engineering firm alleged that Anthropic's Fin AI support chatbot kept him in circular conversations for weeks while he sought a refund for a $3,500 duplicate Claude AI subscription charge. He later sought a chargeback; Anthropic reportedly canceled the working team account, disrupting work and requiring another subscription purchase. The report does not establish that AI caused the charge or cancellation.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI customer support chatbot failed to perform reliably, trapping users in circular loops and failing to escalate billing errors as designed.

Additional risk subdomains

• 5.2 Loss of human agency and autonomy: Anthropic delegated customer support and billing dispute decisions to the AI system, leaving customers with no meaningful human recourse or appeal when errors occurred.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The financial and operational harms were caused by the AI chatbot failing to perform its customer service and escalation functions correctly after being deployed.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The report describes an AI customer service chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to specific transparency obligations to ensure users know they are interacting with an AI.

AI system and alleged parties

• AI system: Fin (Salesforce)
• AI purpose: Chatbot; Sentiment Analysis
• Behaviour type: Assistant
• Alleged developer: Fin, Chatbot developers, AI customer support system developers
• Alleged deployer: Anthropic
• Alleged harmed parties: Engineering firms, Companies, Claude users, AI customer support system users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Psychological

Reported: The report explicitly describes psychological distress and frustration experienced by customers trapped in circular conversations with the AI chatbot.

Directly caused: Customers like Tom reported being 'unbelievably frustrated' by the AI chatbot's inability to resolve billing issues or transfer them to a human.

Indirectly caused: The prolonged billing disputes and lack of human recourse caused significant stress and anxiety for customers managing business accounts.

Inferred additional harm: It is likely that many of the other 560,000 monthly users interacting with the chatbot experienced similar frustration and stress, though specific numbers are not detailed.

People affected

• Occurrences reported: 6
• People reportedly harmed: 6
• People reportedly exposed: 6

Potential causes

Management

• Cost-saving over safety: Anthropic prioritized cheap AI resolutions over maintaining human support.
• Inadequate risk assessment: Management failed to assess risks of AI failures on customer operations.

Technology

• Non-deterministic AI failures: AI agents unpredictably failed to route customer chats to human support.
• Flawed sentiment detection: The chatbot failed to accurately flag frustrated users for escalation.

Data Inputs

• Duplicate account creation: System created duplicate profiles, feeding incorrect billing data to AI.
• Inaccurate billing inputs: Incorrect subscription plans were fed into the AI customer support system.

Human Factors

• Overreliance on AI agents: Anthropic relied almost entirely on AI chatbots to handle billing issues.

Process and Methods

• Flawed escalation rules: Rules failed to properly route complex billing issues to human agents.
• Automated account lockout: Chargeback requests triggered automated systems that shut down active accounts.

Regulatory Environment

• Absence of chatbot laws: No active laws mandated human escalation paths for AI customer service.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, firsthand accounts from multiple named and anonymous sources detailing their specific financial losses and interactions with the AI chatbot. The role of the AI system (Salesforce Fin) and the deployer (Anthropic) is explicitly stated, and the failure modes are well-documented with screenshots and direct quotes.
• Ambiguities identified: The exact technical reason why the AI failed to escalate to humans in these specific cases is not fully detailed, though Tom suggests the non-deterministic nature of the system plays a role.
• Alternative interpretations: The billing errors themselves could be viewed as standard database or software glitches rather than AI failures, but the core of the reported incident is the AI chatbot's failure to resolve or escalate these errors.

Anthropic's customer support chatbot, Fin, failed to resolve billing issues, trapping users in loops and causing minor financial losses and operational disruptions for commercial customers across multiple nations. The incident represents a minor economic and technological security concern related to AI deployment robustness, but poses no threat to national security, critical infrastructure, or sovereignty.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: No clear primary
• Other affected: Canada, United States, Taiwan
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing commercial capability and customer service issue rather than an active national security crisis.
• Autonomy: Human-supervised. The AI chatbot operated autonomously in handling queries and disputes, but existed within a corporate-supervised customer service framework.
• Novelty: Established threat. Chatbot loops, customer service failures, and automated billing glitches are well-known and established issues in technology deployment.

Impact by dimension

• Physical security: Negligible. No physical systems or critical infrastructure were targeted or affected by the customer support chatbot's billing failures.
• Information security: Negligible. The incident involves a customer support billing glitch and does not involve information warfare, intelligence compromise, or espionage.
• Sovereignty: Negligible. This is a commercial customer service issue with no impact on state sovereignty, electoral systems, or government functions.
• Economic security: Minor. Minor financial overcharges and temporary operational disruptions to a few small businesses represent limited economic impact manageable through standard commercial dispute channels.
• Societal stability: Negligible. The incident is limited to customer service frustration and billing disputes, with no implications for societal stability or human rights.