Prominent AI Chatbots Allegedly Produced Incorrect UK Financial and ISA Guidance

Several major AI chatbots, including ChatGPT, Copilot, Gemini, and Meta AI, were reportedly found to have provided incorrect or misleading financial and insurance guidance for UK users. The systems allegedly advised exceeding ISA limits, misstated tax rules, gave wrong travel insurance requirements, and pointed users toward costly refund services.

Risk classification

• Primary risk domain: 3 Misinformation
• Primary risk subdomain: 3.1 False or misleading information

The chatbots generated incorrect and misleading financial, tax, and legal advice, which can lead to inaccurate beliefs and potential material or financial harm for users who rely on them.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The AI systems failed to perform reliably and accurately when queried on domain-specific financial and legal regulations, demonstrating a lack of capability in critical applications.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk is caused by the AI systems generating inaccurate and misleading financial advice, which is an unexpected and unintentional failure mode of the deployed models.

EU AI Act risk tier

• Risk tier: 3 Limited Risk

Limited Risk: The systems described are general-purpose chatbots, which are subject to specific transparency obligations under the EU AI Act to ensure users are aware they are interacting with an AI.

AI system and alleged parties

• AI system: ChatGPT, Copilot, Gemini, Meta AI, Perplexity (Google, Meta, Microsoft, OpenAI, Perplexity AI)
• AI purpose: Chatbot; Question Answering
• Behaviour type: Assistant
• Alleged developer: OpenAI, Microsoft, Meta, Google
• Alleged deployer: OpenAI, Microsoft, Meta, Google
• Alleged harmed parties: Meta AI users, General public of the United Kingdom, General public, Gemini users, Epistemic integrity, Copilot users, ChatGPT users, Chatbot users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: The report explicitly describes epistemic harm where multiple AI chatbots generated false and misleading financial, tax, and legal information.

Directly caused: AI chatbots generated incorrect advice regarding HMRC ISA limits, travel insurance mandates, flight delay compensation, and tax codes, directly misleading users who tested or queried them.

Inferred additional harm: Widespread reliance on these chatbots for critical financial and legal information likely leads to broader dissemination of inaccurate facts and a general erosion of trust in digital information sources.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 2

Potential causes

Management

• Prioritizing Capability over Safety: Deploying general tools for high-stakes tasks without robust safeguards.
• Shifting Verification to Users: Disclaimers used to deflect liability instead of fixing accuracy issues.

Technology

• Hallucination and Logic Errors: Chatbots generated inaccurate tax, insurance, and contract advice.
• Failure to Detect User Errors: Models failed to identify and correct the false 25k ISA allowance premise.

Data Inputs

• Out-of-date Training Data: ChatGPT used an outdated tax code, leading to incorrect calculations.
• Unfiltered Promotional Links: Search results prioritized high-fee premium tax-refund firms over HMRC.

Human Factors

• Overreliance on AI Advice: Consumers blindly trusted chatbot advice for critical financial decisions.
• Lack of User Verification: Users failed to double-check chatbot outputs with official sources.

Process and Methods

• Inadequate Output Verification: Lack of automated accuracy checks for high-risk financial queries.
• Lack of Domain-Specific Guardrails: General-purpose models lack specialized constraints for financial advice.

Regulatory Environment

• Lack of Regulatory Coverage: FCA noted AI advice is not covered by the Financial Ombudsman Service.
• Absence of Chatbot Standards: No strict legal standards governing AI-generated financial recommendations.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, factual details about the testing conducted by Which? and the specific incorrect outputs generated by each chatbot. It also includes first-hand user testimony from a business owner who experienced the issue, making the assessment highly reliable.

Major AI chatbots were found to provide inaccurate and misleading financial, tax, and legal advice to UK consumers. While this poses consumer protection challenges and potential minor personal financial risks, it does not represent a significant threat to national security, critical infrastructure, or state sovereignty.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. This represents an ongoing strategic concern regarding the reliability and accuracy of consumer-facing AI models rather than an active, immediate crisis.
• Autonomy: Human-controlled. The AI systems act as informational assistants, requiring human initiation to prompt and human action to execute any real-world financial or legal decisions.
• Novelty: Established threat. Factual inaccuracies and hallucinations in large language models are well-documented and represent an established technical limitation.

Impact by dimension

• Physical security: Negligible. The incident involves text-based chatbot outputs regarding financial advice and poses no threat to physical systems, critical infrastructure, or human safety.
• Information security: Negligible. There is no indication of active foreign interference, systematic disinformation campaigns, or compromise of classified intelligence in this incident.
• Sovereignty: Negligible. While the chatbots misquoted government rules like HMRC tax codes, this does not threaten state authority, border control, or core government operations.
• Economic security: Minor. The widespread generation of incorrect financial, tax, and legal advice poses a minor risk of aggregate financial loss to consumers, though it remains manageable under standard consumer protection frameworks.
• Societal stability: Negligible. The incident involves isolated consumer advice inaccuracies and does not threaten social cohesion, civil liberties, or population-scale stability.