Pieces Technologies' Clinical AI Systems Allegedly Marketed With Misleading Performance Claims

The Texas Attorney General announced a settlement with Pieces Technologies following allegations that the company misrepresented the accuracy of its healthcare AI systems used for clinical documentation. The state concluded that marketing claims about low error and hallucination rates may have misled hospitals and clinicians relying on the tools in patient care settings. The settlement imposed restrictions on future claims and required greater transparency around performance and risk.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The AI system's actual performance and hallucination rates did not meet the highly accurate metrics advertised, presenting robustness and capability risks in high-risk clinical settings.

Additional risk subdomains

• 3.1 False or misleading information: The generative AI system produces summaries of patient charts that can contain hallucinations, which are false or misleading clinical details.

Causal factors

• Entity: Human
• Intent: Intentional
• Timing: Post-deployment

The incident was primarily caused by human decisions to make deceptive and misleading marketing claims about the AI system's accuracy metrics.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The AI system is used in healthcare as a clinical documentation and decision support tool, which has significant implications for patient safety and health. The EU AI Act classifies AI used in healthcare and medical devices as High Risk.

AI system and alleged parties

• AI system: Pieces Intelligence Platform (Pieces Technologies)
• AI purpose: Health AI Assistant; Writing Assistant
• Behaviour type: Assistant
• Alleged developer: Pieces Technologies, Inc.
• Alleged deployer: Pieces Technologies, Inc.
• Alleged harmed parties: Public health, patients, Healthcare institutions, Epistemic integrity, Clinicians

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Substantial
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: The report explicitly describes epistemic harm in the form of the AI system generating incorrect or misleading clinical documentation (hallucinations) and the company spreading false metrics about these hallucination rates.

Directly caused: The AI system generated clinical summaries and notes that contained undocumented hallucinations, potentially misleading healthcare providers about patient conditions.

Indirectly caused: The dissemination of false accuracy metrics misled hospitals and the public about the reliability of generative AI in clinical settings.

Inferred additional harm: It is likely that some of the 10 million AI-generated clinical documents contained undetected hallucinations that misled clinicians, though the exact number of errors is not specified.

People affected

• Occurrences reported: 1

Potential causes

Management

• Deceptive Marketing Claims: Prioritizing growth led to unsubstantiated safety claims in advertisements.
• Inadequate Risk Assessment: Management failed to assess patient safety risks from inaccurate AI summaries.

Technology

• AI Hallucination Propensity: Generative AI models inherently produce incorrect or misleading clinical text.
• Unverified Accuracy Metrics: Stated hallucination rates of under 0.001 percent were not verified.

Data Inputs

• Real-Time EHR Data Processing: Processing live patient records increased the risk of unverified outputs.
• Lack of Training Data Disclosure: No clear documentation was provided on data used to train the AI models.

Human Factors

• Clinician Overreliance: Medical staff relied on AI summaries without fully understanding limitations.
• Inadequate Operator Training: Lack of guidance for hospital staff on how to monitor for AI inaccuracies.

Process and Methods

• Flawed Metric Calculation: The methodology used to calculate AI hallucination rates was non-transparent.
• Lack of Independent Auditing: No objective third-party testing was conducted to substantiate AI claims.
• Deficient Risk Documentation: Failure to document known harmful misuses and limitations of the AI tool.

Regulatory Environment

• Absence of Federal AI Laws: Lack of proactive federal regulations governing clinical AI safety standards.
• Post-Deployment Enforcement: Regulators relied on retrospective consumer protection laws to address risks.

Information quality

• Classification confidence: High
• Reason for confidence: The reports from legal and regulatory sources provide clear, consistent details about the Texas AG investigation, the specific deceptive claims made by Pieces Technologies, and the terms of the settlement. There is no conflicting information regarding the core facts of the deceptive marketing allegations.
• Ambiguities identified: The reports do not specify the actual hallucination rate of the AI system, only that the advertised rate of <0.001% was likely inaccurate.
• Alternative interpretations: None. The legal nature of the settlement and the allegations of deceptive trade practices are straightforward.

The Texas Attorney General settled with Pieces Technologies over deceptive marketing claims regarding the accuracy of its clinical generative AI tool. While posing potential risks within the healthcare critical infrastructure sector due to unverified hallucination rates, the incident was resolved legally and had negligible direct impact on national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Pieces Technologies

Threat characteristics

• Imminence: Long-term. The immediate regulatory issue is resolved through a settlement, but it represents an ongoing strategic concern regarding AI safety validation.
• Autonomy: Human-supervised. The AI generates clinical documentation to assist healthcare providers, who remain responsible for final clinical decisions and oversight.
• Novelty: Established threat. Deceptive marketing and generative AI hallucinations are well-established risks, though this marks an early state-level regulatory enforcement action.

Impact by dimension

• Physical security: Minor. The tool was deployed in healthcare, a critical infrastructure sector, with inaccurate accuracy claims that posed potential risks to patient safety, though no actual physical harm was reported.
• Information security: Negligible. No information warfare, intelligence compromise, or systematic foreign disinformation campaigns were associated with this incident.
• Sovereignty: Negligible. No threat to state authority or core government operations; the incident was resolved through standard state-level regulatory enforcement.
• Economic security: Negligible. The incident involves deceptive marketing by a domestic private firm and does not threaten national economic stability or strategic technological competitiveness.
• Societal stability: Negligible. No widespread societal destabilization, mass surveillance, or systemic civil rights violations occurred as a result of this incident.