Security Robot Rolls Over Child in Mall

On July 7, 2016, a Knightscope K5 autonomous security robot collided with a 16-month old boy while patrolling the Stanford Shopping Center in Palo Alto, CA.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The robot failed to perform reliably under normal operating conditions by failing to detect and avoid a small child in its path.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The physical collision was caused by the autonomous navigation decisions of the deployed K5 robot, which unintentionally failed to detect and avoid the toddler.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The K5 is an autonomous security robot deployed in public spaces with direct physical safety implications, which fits the definition of systems having significant implications for safety and public interests.

AI system and alleged parties

• AI system: K5 Autonomous Data Machine (Knightscope)
• AI purpose: Smart Surveillance; Threat Detection
• Behaviour type: Autonomous
• Alleged developer: Knightscope
• Alleged deployer: Stanford Shopping Center
• Alleged harmed parties: Child

Harm severity

Highest direct severity in any category: Negligible. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 1

Potential causes

Management

• Unaddressed Prior Incidents: A similar incident days earlier was mentioned but not investigated.
• Prioritizing Cost over Safety: Deploying robots as cheap security labor before resolving navigation bugs.

Technology

• Sensor Blind Spots: Sensors failed to detect the small stature of the toddler.
• No Vibration Alert Triggered: The machine failed to register the collision via its vibration sensors.
• Lack of Collision Shutdown: The motors did not fault and the robot kept moving forward after impact.

Data Inputs

• Inadequate Object Classification: The small size of the toddler confused the robot's sensor fusion algorithms.
• Lack of Impact Sensor Data: No physical feedback was registered to indicate an obstacle was run over.

Human Factors

• Unpredictable Toddler Movement: The child ran towards the robot or backwards into its path.
• No Warning Labels on Machine: The robot lacked signs warning parents to keep children away.
• Attractive Friendly Appearance: The R2-D2-like design attracted the child to approach the machine.

Process and Methods

• Insufficient Public Space Testing: The system was not fully validated for navigating around small children.
• No External Manual Override: The parent tried to push the robot but could not manually stop it.

Regulatory Environment

• Lack of Autonomous Bot Standards: No regulations governed the safety of heavy security robots in public spaces.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent core facts about the collision, the parties involved, the location, and the resulting minor injuries. While there is a slight conflict between the mother's account and the company's account regarding the exact dynamics of the collision, the physical outcome and the failure of the robot's sensors to prevent the collision are clear.
• Ambiguities identified: There is a conflict between Knightscope's claim that the robot veered and stopped versus the mother's claim that the robot struck the child from behind and kept moving.
• Alternative interpretations: Knightscope suggested the child ran backwards into the robot, framing it as a 'freakish accident' rather than a system failure.

In July 2016, an autonomous Knightscope K5 security robot collided with and minorly injured a toddler at a California shopping center. While highlighting early public safety challenges of integrating physical AI systems into crowded pedestrian environments, the incident was an unintentional localized accident with negligible national security implications.

• Overall national security impact: Negligible
• Response level: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident was a localized, past event that does not represent an active crisis or near-term threat to national security.
• Autonomy: Full autonomy. The Knightscope K5 robot navigated and patrolled the shopping center autonomously without direct human control or real-time supervision.
• Novelty: First-of-its-kind. One of the earliest publicly documented incidents of an autonomous physical security robot colliding with and injuring a pedestrian in a public space.

Impact by dimension

• Physical security: Negligible. The incident involves a localized minor collision between a commercial security robot and a toddler in a shopping mall, presenting no threat to national critical infrastructure or physical security.
• Information security: Negligible. No elements of information warfare, espionage, or intelligence compromise were present in this localized commercial accident.
• Sovereignty: Negligible. The incident was a local public safety occurrence with no impact on government operations, elections, or state sovereignty.
• Economic security: Negligible. While the incident led to temporary suspension of the robot model at the mall, it had no significant impact on strategic industries, national economic stability, or technological competitiveness.
• Societal stability: Negligible. The accident caused minor physical injury to one child, with no implications for mass surveillance, systemic human rights violations, or large-scale societal stability.