SN Technologies Reportedly Lied to a New York State School District about Its Facial and Weapon Detection Systems’ Performance

SN Technologies allegedly misled Lockport City Schools about the performance of its AEGIS face and weapons detection systems, downplaying error rates for Black faces and weapon misidentification.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The AEGIS system demonstrated significant racial bias, misidentifying Black students at much higher rates (4x for Black men and 16x for Black women) than white students, resulting in unequal performance across demographic groups.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The weapon detection system frequently failed to perform reliably, mistaking broom handles for guns and requiring multiple recalibrations.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The primary risks and harms arose from the AI system's biased facial recognition performance and false weapon detections after it was deployed in January 2020.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is a facial recognition and surveillance tool deployed in an educational institution, which falls under high-risk applications affecting fundamental rights and safety.

AI system and alleged parties

• AI system: AEGIS (SN Technologies)
• AI purpose: Face Recognition; Threat Detection
• Behaviour type: Assistant
• Alleged developer: SN Technologies
• Alleged deployer: Lockport City School District
• Alleged harmed parties: Black students

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Minor
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Minor, indirect Negligible
• Psychological: direct Negligible, indirect Minor
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Financial

Reported: The report explicitly describes indirect financial losses through the misallocation of public funds, though no specific dollar amount is quantified.

Directly caused: The report does not specify a dollar amount for the contract, resulting in an average reported direct loss of 0 per occurrence.

Indirectly caused: The school district spent Smart Schools state funding on the surveillance system instead of educational upgrades like laptops and internet connectivity.

Inferred additional harm: The school district likely spent hundreds of thousands of dollars in public funds on a flawed system that was ultimately rendered ineffective by mask mandates.

Differential treatment

Reported: The report explicitly describes differential treatment of individuals or groups caused directly by the incident.

Directly caused: The AEGIS facial recognition system performed with significant racial bias, misidentifying Black men 4 times more often and Black women 16 times more often than white men.

Indirectly caused: Black students are disproportionately subjected to the risk of false positives and potential hostile police interventions.

Inferred additional harm: Increased surveillance and systemic bias against minority students within the school district, leading to unequal educational environments.

Civil rights

Reported: The report explicitly describes violations of human or civil rights caused directly by the incident.

Directly caused: Violation of students' privacy and civil liberties through biased surveillance, leading to a lawsuit by the NYCLU on behalf of parents.

Indirectly caused: Potential infringement on the right to a safe and equal education free from discriminatory surveillance.

Inferred additional harm: Broader chilling effect on student expression and movement due to constant, biased surveillance.

Privacy

Reported: The report explicitly describes privacy violations caused directly by the deployment of the system.

Directly caused: The deployment of a facial recognition network in school buildings tracking students' biometric data.

Inferred additional harm: Unauthorized or invasive collection of biometric data of minors without robust consent or safeguards.

Psychological

Reported: The report explicitly describes psychological distress and anxiety among parents and students caused indirectly by the incident.

Indirectly caused: The deployment of the biased system caused significant outrage, fear, and distress among Black parents and students who worried about potential armed police responses to false alarms.

Inferred additional harm: Widespread anxiety among minority students who feel targeted or unsafe due to the biased surveillance system operating in their schools.

Epistemic

Reported: The report explicitly describes epistemic harm caused directly by the developer's misrepresentations.

Directly caused: SN Technologies misrepresented NIST test results, claiming their algorithm ranked 49th out of 139 and downplaying the bias, which misled the school district into purchasing the system.

Inferred additional harm: Erosion of trust in educational technology procurement and public decision-making processes due to corporate deception.

People affected

• Occurrences reported: 1
• People reportedly harmed: 300
• People reportedly exposed: 3000

Potential causes

Management

• Procurement Based on Misinformation: School board purchased the system based on misleading vendor claims.
• Failure to Address Stakeholder Fears: District management ignored parent and civil liberties groups concerns.
• Inadequate Risk Assessment: Management failed to assess risks of false weapon alerts triggering police.

Technology

• High Racial Bias in Algorithm: The licensed id3 algorithm misidentified Black students at much higher rates.
• Poor Object Discrimination: The computer vision system regularly mistook broom handles for weapons.
• Sensitivity to Camera Disturbance: School construction disturbed cameras, requiring frequent recalibrations.

Data Inputs

• Biased Training Data: Underlying facial recognition model performed worse on non-white students.
• Obscured Visual Inputs: Pandemic mask-wearing rendered the facial recognition inputs ineffective.
• Uncalibrated Weapon Profiles: Lack of diverse negative weapon training examples led to false gun alerts.

Human Factors

• Vendor Misrepresentation of Bias: The vendor CEO misled school officials about the algorithm's actual bias rates.
• Fear of Armed Police Escalation: Parents feared false alerts would trigger aggressive, tragic police reactions.
• Overreliance on Automated Alerts: Initial system design used automated alerts before adding human verification.

Process and Methods

• Inadequate Independent Auditing: The district relied on vendor claims before an audit revealed the true bias.
• Lack of Pre-deployment Testing: The system was deployed in schools without resolving basic detection errors.
• Flawed Accuracy Calculations: NIST scientists described vendor accuracy claims as completely nonsensical.

Regulatory Environment

• Weak State Oversight: NYSED abruptly approved the system after initially opposing its deployment.
• Absence of School AI Bans: Lack of timely legislation allowed districts to deploy unvetted AI systems.
• Unregulated Funding Allocation: Smart Schools funding was approved for surveillance without safety guidelines.

Information quality

• Classification confidence: High
• Reason for confidence: The report is highly detailed, citing internal emails, independent audit reports, and statements from NIST scientists, school board members, and the developer.
• Ambiguities identified: The exact financial cost of the system and whether any false alerts led to actual police dispatches are not specified.
• Alternative interpretations: The developer argues the system is 99.9% accurate and that the issues were minor calibration adjustments, contrasting with the critics' claims of systemic failure.

The deployment of the AEGIS surveillance system in New York schools highlights civil rights and privacy concerns from biased AI tools. While the system exhibited significant racial bias and high false-positive rates, its impact is localized to school administration and civil rights litigation, presenting minimal direct threat to national security.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: SN Technologies

Threat characteristics

• Imminence: Long-term. Represents an ongoing regulatory and civil rights concern regarding public surveillance rather than an active crisis.
• Autonomy: Human-supervised. The system was designed to flag potential matches for human review, though it operated autonomously in scanning video feeds.
• Novelty: Evolved capability. Combines facial recognition with weapon detection in a public school setting, representing an evolution of established biometric surveillance concerns.

Impact by dimension

• Physical security: Negligible. No direct threat to critical infrastructure, kinetic systems, or physical security. System malfunctioned locally in a school environment with no physical damage.
• Information security: Negligible. The incident involves local school surveillance and does not impact national intelligence capabilities, classified models, or information warfare.
• Sovereignty: Minor. Local educational governance was misled by vendor misrepresentations, leading to state legislative intervention, but does not threaten core national sovereignty or state authority.
• Economic security: Negligible. Involves local school district funding misallocation rather than threats to strategic industries, national economic stability, or critical supply chains.
• Societal stability: Minor. Biased surveillance system deployed in public schools violated student privacy and disproportionately targeted minority students, leading to civil rights lawsuits and localized public distress.