Companies × risks
Overreliance and unsafe use
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
7 of the world's largest companies we reviewed acknowledge the AI risk "Overreliance & unsafe use", including Microsoft, IBM and Salesforce.
Which companies acknowledge this risk?
- Microsoft: “Be aware of the risk of overreliance: Overreliance on AI happens when users accept incorrect or incomplete AI outputs... For the end-user, overreliance could result in decreased productivity, loss of trust, product abandonment, financial loss, psychological harm, physical harm, among others. (for example, a doctor accepts an incorrect AI output).” (Transparency Note for Microsoft 365 Copilot)
- IBM: “Over-reliance occurs when a person puts too much trust in a model... accepting a model's output when the model's output is likely incorrect... over/under reliance can lead to poor decision making because of the misplaced trust in the AI system, with negative consequences.” (Foundation models: Opportunities, risks and mitigations)
- Salesforce: “Automation bias, where users overly trust AI outputs without critical analysis, risks amplifying errors and ethical concerns.” (Salesforce Trusted AI Impact Report)
- Palantir: “They should also take account of the propensity of such models to induce levels of end-user trust that exceed the system's competence.” (Palantir Australia Response to Supporting Responsible AI ...)
- SAP: “With GenAI, human error can extend to misplaced trust in generated output. Adversaries can take advantage of this trust...” (SAP Generative AI Cybersecurity Strategy)
- L'Oreal: “WE WILL NOT ... Over rely on algorithmic decision-making without safeguards and human oversight.” (L'Oreal - CODE OF ETHICS)
- National Australia Bank: “...the overreliance on algorithmic outcomes without adequate explainability, transparency or appreciation of the impacts.” (NAB H1 2025 Offering Circular)