“Amazon’s Choice” Algorithm Failed to Recommend Functional Products and Prone to Review Manipulation

Amazon’s “Amazon’s Choice” algorithm recommended poor-quality defective products and were reportedly susceptible to manipulation by inauthentic reviews.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The recommendation algorithm fails to perform robustly when faced with manipulated or recycled reviews, leading it to recommend low-quality or faulty products to users.

Additional risk subdomains

• 3.1 False or misleading information: The algorithm generates misleading recommendations (the 'Amazon's Choice' badge) that misrepresent product quality to consumers.
• 4.3 Fraud, scams, and targeted manipulation: Third-party sellers exploit the recommendation system through review manipulation to gain an unfair financial advantage.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk of recommending shoddy products is an unexpected outcome of the Amazon's Choice recommendation algorithm pursuing its goal of highlighting highly-rated products.

EU AI Act risk tier

• Risk tier: 4 Minimal or No Risk

Minimal or No Risk: The Amazon's Choice algorithm is a product recommendation system, which represents a low-risk application with no additional regulatory obligations under the EU AI Act.

AI system and alleged parties

• AI system: Amazon’s Choice algorithm (Amazon)
• AI purpose: Product Recommendation; Content Curation
• Behaviour type: Assistant
• Alleged developer: Amazon
• Alleged deployer: Amazon
• Alleged harmed parties: Amazon users

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Substantial
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Epistemic

Reported: Yes, the report describes the erosion of trust in reviews and the 'Amazon's Choice' badge, making it harder for shoppers to distinguish legitimately good products from bad ones.

Directly caused: The recommendation algorithm directly generated misleading recommendations by labeling manipulated products as 'Amazon's Choice', spreading false signals of quality to users.

Indirectly caused: The deletion of genuine negative reviews and the persistence of fake positive reviews corrupted the information ecosystem of the marketplace.

Inferred additional harm: Widespread erosion of consumer trust in online review systems and e-commerce recommendations at scale.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Prioritizing Profit Over Safety: Amazon profits from sales driven by the misleading badge, reducing incentive.
• Inadequate Risk Mitigation: Management failed to detail action plans when badge abuse is discovered.

Technology

• Flawed Recommendation Algorithm: Algorithm recommends items based on star scores without detecting fraud.
• Inability to Detect Nuance: The algorithm cannot detect nuances in review manipulation or sentiment.
• Lack of Algorithmic Accountability: It is harder to hold an automated algorithm accountable than human reviewers.

Data Inputs

• Recycled and Fake Reviews: Sellers reuse reviews from other products to inflate ratings.
• Deletion of Negative Reviews: Negative reviews are removed, distorting the overall star rating data.
• Misleading Reference Prices: Inflated list prices manipulate the apparent savings for consumers.

Human Factors

• Overreliance on the Badge: Shoppers trust the Amazon's Choice badge blindly without verifying reviews.
• Seller Manipulation of Ratings: Sellers offer replacements to coerce customers into revising negative reviews.
• Blind Trust in Amazon's Brand: Customers treat the badge as a trusted-brand seal of approval.

Process and Methods

• Inadequate Review Verification: Amazon lacks robust processes to verify review authenticity before badging.
• Poor Handling of Reports: Amazon failed to follow up on user reports of deleted reviews.
• Lack of Manual Oversight: No human review team verifies products before assigning Amazon's Choice.

Regulatory Environment

• Lack of Regulatory Enforcement: Regulators fail to actively police and penalize review recycling practices.
• Deceptive Advertising Patterns: Weak enforcement allows systemic deceptive reference pricing to persist.

Information quality

• Classification confidence: High
• Reason for confidence: The report clearly outlines the mechanism of the failure (review manipulation leading to bad recommendations), identifies the specific AI system (Amazon's Choice), and provides a concrete user case (R.D.) alongside expert commentary.
• Ambiguities identified: The exact technical details of how the Amazon's Choice algorithm weighs star ratings versus other factors are not fully disclosed.
• Alternative interpretations: The incident could be viewed purely as a human fraud issue (seller review manipulation) rather than an AI system failure, though the algorithm's uncritical reliance on these metrics makes it an AI safety/robustness issue.

Amazon's 'Amazon's Choice' recommendation algorithm was manipulated by third-party sellers using fake and recycled reviews to promote low-quality products. While this presents consumer protection and deceptive advertising issues under FTC purview, it has negligible implications for national security.

• Overall national security impact: Negligible
• Scope: Multiple nations
• Primary target: United States
• Other affected: Unknown
• Alleged perpetrator: Third-party sellers

Threat characteristics

• Imminence: Long-term. Represents an ongoing, persistent commercial marketplace vulnerability rather than an active national security crisis.
• Autonomy: Human-supervised. The recommendation algorithm automatically assigns badges based on data inputs, but operates under Amazon's overall platform management and policy enforcement.
• Novelty: Established threat. Review manipulation, astroturfing, and algorithmic exploitation are well-documented, long-standing issues in e-commerce.

Impact by dimension

• Physical security: Negligible. The incident involves consumer product recommendations and has no impact on physical systems, critical infrastructure, or human safety.
• Information security: Negligible. Review manipulation is limited to commercial e-commerce products and does not compromise intelligence capabilities or constitute strategic information warfare.
• Sovereignty: Negligible. The failure of a commercial e-commerce recommendation system has no bearing on state authority, borders, or core government operations.
• Economic security: Negligible. While causing minor financial losses to individual consumers, it does not threaten national economic stability, strategic industries, or critical supply chains.
• Societal stability: Negligible. The erosion of consumer trust in online reviews is a marketplace issue and does not pose a threat to social cohesion, civil liberties, or population safety.