Sound Intelligence's Aggression Detector Misidentified Innocuous Sounds

Sound Intelligence's "aggression detection" algorithm deployed by schools reportedly contained high rates of false positive, misclassifying laughing, coughing, cheering, and loud discussions.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The aggression detection algorithm exhibited unequal performance across demographic groups, specifically showing a higher rate of false positives for female students due to their higher-pitched voices.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The system demonstrated a significant lack of robustness, frequently generating false positives for benign sounds like coughing and failing to detect actual screaming.
• 5.1 Overreliance and unsafe use: Schools and hospitals relied on the system for critical security and safety monitoring, despite its inability to reliably distinguish aggression from normal human behavior.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The failures of the aggression detection system to accurately classify sounds occurred post-deployment due to unintentional algorithmic limitations.

EU AI Act risk tier

• Risk tier: 2 High Risk

Risk Level 2: High Risk. The system is deployed in educational institutions (schools) and healthcare facilities (hospitals) to monitor safety and security, which directly impacts fundamental rights and public interests.

AI system and alleged parties

• AI system: aggression detection algorithm (Sound Intelligence)
• AI purpose: Threat Detection; Smart Surveillance
• Behaviour type: Assistant
• Alleged developer: Sound Intelligence
• Alleged deployer: Rock Hill Schools, Pinecrest Academy Horizon
• Alleged harmed parties: students, Rock Hill School students, Pinecrest Academy Horizon students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Minor, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Substantial, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: Yes, the report describes financial expenditures on the system, including a hospital pilot program.

Directly caused: The Valley Hospital spent $22,000 on a three-year pilot program for the detector, which they are now phasing out due to its ineffectiveness.

Indirectly caused: Schools and other facilities purchased microphones preloaded with the software, with one Louroe microphone selling for about $1,000.

Inferred additional harm: Other schools and hospitals among the hundreds of customers likely wasted thousands of dollars purchasing and maintaining this ineffective technology.

Differential treatment

Reported: Yes, the report explicitly describes differential treatment of female students due to voice pitch.

Directly caused: The algorithm disproportionately triggered false positives for female students, as every speaking voice in the pair testing that triggered the algorithm belonged to a female student due to their higher pitch.

Indirectly caused: Female students are subjected to higher rates of false accusations of aggression and subsequent security scrutiny compared to male students.

Inferred additional harm: Female students across all deployed school environments likely face disproportionate surveillance and disciplinary risks due to the system's gender bias.

Civil rights

Reported: Yes, the report describes concerns regarding student privacy and the right to be free from unwarranted surveillance.

Directly caused: The microphones allow school administrators to record, replay, and store snippets of students' conversations indefinitely, infringing on their privacy rights.

Inferred additional harm: Thousands of students' private conversations in public school spaces are recorded and stored without explicit consent, violating their civil liberties.

Privacy

Reported: Yes, the report describes privacy concerns related to the recording and storage of audio snippets.

Directly caused: The system's microphones allow administrators to record, replay, and store snippets of student conversations indefinitely.

Inferred additional harm: Mass surveillance of students, patients, and workers in public spaces without their knowledge or consent, leading to unauthorized data retention.

Psychological

Reported: Yes, the report describes potential psychological impacts such as student distrust and alienation.

Indirectly caused: An associate professor of psychiatry noted that the audio surveillance could have the unintended consequence of increasing student distrust and alienation.

Inferred additional harm: Widespread anxiety and stress among students and staff who are constantly surveilled by a system prone to false positives.

People affected

• Occurrences reported: 1
• People reportedly harmed: 10
• People reportedly exposed: 760

Potential causes

Management

• Misleading Marketing Claims: Device marketed as school safety solution despite known failure modes.
• Premature Commercialization: Selling system to schools and hospitals before resolving high error rates.

Technology

• Context-Blind Algorithm: Algorithm uses audio features without understanding words, meaning, or context.
• Acoustic Feature Correlation: Algorithm equates aggression with high pitch and rough or strained vocal tones.
• Biased Feature Sensitivity: Higher-pitched female voices are more prone to triggering false positives.

Data Inputs

• Unrepresentative Training Data: Training data lacked high-pitched shrieks or school-specific vocalizations.
• Incorrect Data Labeling: Some shrieks were labeled as non-aggressive due to similarity to baby cries.
• Out-of-Context Calibration: Calibrated on European pub data but used in schools and hospitals.

Human Factors

• Overreliance on Automation: Users substitute algorithmic alerts for human observation and judgment.
• Misunderstanding System Limits: School officials expected the device to detect quiet, non-strained aggression.

Process and Methods

• Inadequate Local Calibration: Fine-tuning sensitivity to stop false alarms increases risk of false negatives.
• Flawed Performance Testing: System wasn't comprehensively tested across diverse demographic voice ranges.

Regulatory Environment

• Lack of Algorithmic Auditing: No standards require independent auditing of public safety algorithms.

Information quality

• Classification confidence: High
• Reason for confidence: The ProPublica reports provide extensive empirical testing data, methodology, and direct interviews with the AI developer's CEO, school administrators, and hospital security chiefs, offering a highly detailed and credible account of the system's performance and real-world impacts.
• Ambiguities identified: None significant, though the exact proprietary features used by Sound Intelligence's algorithm are not fully disclosed, ProPublica successfully reconstructed closely correlating features for their analysis.
• Alternative interpretations: The developer argues the system is a valuable early warning system despite its imperfections, whereas critics view it as an unreliable and invasive surveillance tool.

The Sound Intelligence aggression detection system's high false-positive rates, failure to detect actual threats, and gender bias in schools and hospitals represent a minor national security concern. While highlighting vulnerabilities in public space AI surveillance and civil rights issues, the impact remains localized and manageable through standard procurement and security procedures.

• Overall national security impact: Minor
• Scope: Multiple nations
• Primary target: United States
• Other affected: Unknown
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. Represents an ongoing, long-term policy and technology robustness challenge rather than an imminent national crisis.
• Autonomy: Human-supervised. The AI acts as an assistant by flagging potential aggression, but human operators must verify alerts and make final security decisions.
• Novelty: Established threat. Algorithmic bias and high false-positive rates in automated sensor and surveillance systems are well-documented, established issues.

Impact by dimension

• Physical security: Minor. System deployed in schools and hospitals failed to detect actual threats and triggered false alarms, causing minor disruption but manageable under standard local security.
• Information security: Negligible. No evidence of information warfare, intelligence compromise, or state-sponsored disinformation campaigns associated with this incident.
• Sovereignty: Negligible. No impact on national sovereignty, border control, or core federal government decision-making processes.
• Economic security: Negligible. Minor financial loss for local institutions purchasing the equipment (e.g., USD 22,000 hospital pilot), but no strategic economic or technological threat.
• Societal stability: Minor. Raises local privacy and civil liberties concerns due to continuous audio recording and gender-based bias, but lacks scale to threaten national societal stability.