Year-long AI Surveillance Pilot in Two South Australian Aged Care Facilities Reportedly Overwhelmed Staff with False Positives

Between March 2021 and March 2022, an AI-enabled video and audio monitoring system was trialed in two South Australian aged care facilities. According to an independent audit commissioned by South Australia Health, the system produced over 12,000 false alerts, overwhelming staff and contributing to at least one missed real incident. The report concluded the system did not reach accuracy levels acceptable to staff or management during the pilot.

Risk classification

• Primary risk domain: 7 AI system safety, failures, & limitations
• Primary risk subdomain: 7.3 Lack of capability or robustness

The primary failure was the AI system's lack of capability and robustness, as it failed to perform reliably by generating 12,000 false alerts and missing a real fall.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: The system was deployed for critical safety monitoring in an aged care setting before it was robust enough, leading to unsafe conditions when staff became overwhelmed.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The incident was caused by the AI surveillance system generating excessive false alerts, which was an unintentional outcome of its deployment.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk (Level 2): The system is an AI-based safety and monitoring tool deployed in a residential care/healthcare setting, which has significant implications for the safety and physical well-being of vulnerable individuals.

AI system and alleged parties

• AI system: AI-enabled video and audio monitoring system
• AI purpose: Smart Surveillance; Threat Detection
• Behaviour type: Autonomous
• Alleged developer: Sturdie Trade Services
• Alleged deployer: South Australia Health
• Alleged harmed parties: Residents and staff of Northgate House, Residents and staff of Mount Pleasant District Hospital, Northgate House, Mount Pleasant District Hospital

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Negligible, indirect Negligible
• Civil rights: direct Negligible, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report describes a trial costing AU $785,000 (approx. £467,000) that was ultimately deemed inconclusive and botched.

Directly caused: The trial of the failed system cost AU $785,000 of federal funding. Average loss per occurrence (1 trial/pilot): AU $785,000.

Inferred additional harm: Additional indirect costs may include the time and resources spent by staff managing the false alerts and the cost of subsequent reviews (e.g., PwC audit).

Psychological

Reported: The report describes staff experiencing alert fatigue and being overwhelmed by the workload associated with responding to alerts.

Indirectly caused: Aged care staff experienced severe alert fatigue and stress from responding to 12,000 false alerts.

Inferred additional harm: The resident who fell and was not assisted likely experienced significant distress, anxiety, or fear while waiting for help.

People affected

• Occurrences reported: 1
• People reportedly harmed: 1
• People reportedly exposed: 57

Potential causes

Management

• Unrealistic Alert Expectations: SA Health anticipated only 10 false alerts per day, underestimating the risk.
• Botched Rollout Management: Poor coordination and rollout of the trial as noted by the Health Minister.

Technology

• Inability to Distinguish Objects: AI struggled to separate inanimate objects like coats from actual people.
• Overly Sensitive Audio Tuning: System triggered alerts on normal high ambient noise levels in facilities.
• Misclassification of Crouch Motions: Staff crouching to help residents was misidentified as resident fall events.

Data Inputs

• Poor Alignment of Action Patterns: Programmed movement and sound patterns did not match actual resident behavior.
• Environmental Clutter and Occlusion: Multiple people or moving furniture blocked cameras or confused the AI model.

Human Factors

• Severe Alert Fatigue: Staff stopped responding to alerts due to overwhelming number of false alarms.
• Workforce Pressures: Existing staff shortages made responding to constant false alerts impossible.

Process and Methods

• Underestimated Training Time: The trial underestimated the time required to train AI in an aged care context.
• Inadequate Contextual Testing: Lack of comprehensive testing before launching the pilot across the sites.
• Change in Technology Provider: Original monitoring company pulled out, leading to a botched rollout.

Information quality

• Classification confidence: High
• Reason for confidence: The reports are highly consistent, drawing from an independent PwC audit of the trial. They provide specific details on the number of false alerts (12,000), the duration (12 months), the cost (AU $785,000), and the specific consequences (alert fatigue and one missed fall).
• Ambiguities identified: The exact physical consequences or injuries of the missed fall are not detailed.
• Alternative interpretations: None. The failure of the system and its direct link to alert fatigue and the missed fall are clearly documented.

A pilot trial of an AI surveillance system in Australian aged care facilities failed due to excessive false alerts, leading to alert fatigue and one missed resident fall. This represents a localized technology failure with negligible national security implications.

• Overall national security impact: Negligible
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The incident involves a concluded 12-month pilot trial, representing no immediate or near-term national security crisis.
• Autonomy: Human-supervised. The system generated and sent alerts autonomously, but relied on human care staff to make decisions and intervene.
• Novelty: Established threat. False alerts and subsequent sensor/alarm fatigue are well-documented, established issues in automated systems.

Impact by dimension

• Physical security: Negligible. The failure of a localized pilot surveillance system in two aged care facilities does not threaten national physical security or critical infrastructure.
• Information security: Negligible. No elements of information warfare, intelligence compromise, or systematic manipulation are present in this incident.
• Sovereignty: Negligible. While funded by the federal government, the failure of a localized pilot project does not threaten state authority, territorial control, or core government operations.
• Economic security: Negligible. The financial loss of AU 785,000 is negligible on a national scale and does not threaten strategic industries or overall economic stability.
• Societal stability: Negligible. While raising minor privacy and care quality concerns locally, the incident does not pose a threat to large-scale social cohesion, civil liberties, or population safety.