Algorithm Assessing Risk Faced by Victims of Gender Violence Misclassified Low-Risk Cases, Allegedly Leading to Homicide of Women and Children in Spain

In Spain, the algorithm that assesses recidivism risk in gender violence, VioGén, have critically underestimated the level of risk in a series of cases that ended in homicide of women and children since its first deployment.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The system exhibits significant demographic bias, systematically assigning lower risk scores to women without children, which resulted in a 44% recall difference in extreme risk cases.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The system fails to perform reliably, frequently underestimating risk and failing to predict lethal violence, leading to the murder of unprotected victims.
• 5.1 Overreliance and unsafe use: Police officers blindly rely on the algorithmic score in 95% of cases, treating the recommender system as an automated decision-maker without exercising meaningful human oversight.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The physical and psychological harms resulted from the VioGén algorithm's failure to accurately assess risk after it was deployed in police stations.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is a law enforcement risk assessment tool used by public authorities to evaluate the safety and fundamental rights of domestic violence victims.

AI system and alleged parties

• AI system: VioGén
• AI purpose: Recidivism Prediction; Resource Allocation
• Behaviour type: Assistant
• Alleged developer: Spanish Secretary of State for Security, Spanish Ministry of Interior
• Alleged deployer: Spanish Ministry of Interior
• Alleged harmed parties: Spanish victims of gender violence

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Severe
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Substantial
• Civil rights: direct Negligible, indirect Substantial
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Minor, indirect Substantial
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Physical

Reported: The reports explicitly describe multiple fatalities of women and children who were murdered after the system underestimated their risk.

Indirectly caused: The system's underestimation of risk indirectly led to the murder of at least 126 women in the quantitative study and children like Nerea (6) and her sibling (2) due to the denial of police protection.

Inferred additional harm: Given that VioGén has processed over 600,000 cases with a high rate of false negatives, it is highly likely that hundreds of other women and children have suffered physical injuries or death due to inadequate protection.

Differential treatment

Reported: The reports explicitly describe systematic differential treatment of women based on whether they have children.

Directly caused: The algorithm systematically assigned lower risk scores to women without children, resulting in a 44% recall difference in extreme risk cases.

Indirectly caused: Women without children were disproportionately denied police protection, leaving them highly vulnerable to further abuse and homicide.

Inferred additional harm: Thousands of childless women seeking protection were systematically subjected to higher risk of violence due to this algorithmic bias.

Civil rights

Reported: The reports describe failures to protect victims' lives and safety, which are fundamental human rights.

Indirectly caused: The system's failure to provide adequate protection violated the victims' fundamental right to life, physical integrity, and equal protection under the law, as mandated by the Istanbul Convention.

Inferred additional harm: The systemic denial of protection likely violated the civil and human rights of thousands of domestic violence survivors across Spain.

Psychological

Reported: The reports explicitly describe severe distress, anxiety, and trauma experienced by victims during and after the assessment process.

Directly caused: The rigid, binary questionnaire and lack of support during the police interview caused direct distress, with victims describing it as a 'nightmare' and feeling 'lost' and 'unprotected'.

Indirectly caused: The lack of police protection left victims exposed to ongoing threats, harassment, and abuse, causing severe psychological trauma.

Inferred additional harm: It is likely that thousands of the 600,000 women who went through the system experienced severe anxiety and trauma due to being denied protection and left vulnerable to their abusers.

People affected

• Occurrences reported: 1
• People reportedly harmed: 128
• People reportedly exposed: 600000

Potential causes

Management

• Budget-Driven Risk Calibration: Management scales back protective risk scores to match police funding cuts.
• Rejection of External Audits: The Ministry of Interior repeatedly denied independent external audit requests.
• Decoupled Technical Evolution: The transition to machine learning lacks public debate and research oversight.

Technology

• Resource-Constrained Risk Tuning: Algorithm limits extreme risk scores based on available police funding.
• Opaque Algorithmic Formula: The exact formula and weights used to calculate risk remain undisclosed.
• Rigid Binary Output System: The system relies on binary inputs and lacks flexibility for nuance.

Data Inputs

• Bias Against Childless Women: Victims without children are systematically assigned lower risk scores.
• Underestimation of Non-Physical Abuse: The system prioritizes physical violence over psychological or digital abuse.
• Compromised Input Data Quality: Victims in shock or under duress provide incomplete or distorted answers.

Human Factors

• Automation Bias in Police Officers: Officers blindly stick to the algorithm's risk score in 95% of cases.
• Lack of Specialized Gender Training: Police officers administering the test lack training in gender-based violence.
• Victim Cognitive Overload: Traumatized victims struggle to recall details under pressure from officers.

Process and Methods

• Rigid and Inflexible Questionnaires: Fixed question formats do not allow victims to provide necessary explanations.
• No End-User Engagement: Victims and women's organizations were never consulted during system design.
• Exclusion of Aggressor Assessments: Risk is calculated without conducting any direct evaluation of the aggressor.

Regulatory Environment

• Delayed Istanbul Convention Alignment: System forms were not updated to comply with the treaty until years later.
• Unenforced Interdisciplinary Mandate: The 2004 law requiring psychologist and social worker teams was not enforced.
• Lack of Independent Oversight: No public or independent entity has regular access to audit VioGen data.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide extensive, detailed information from independent audits, academic researchers, and official government responses, allowing for a highly reliable classification.
• Ambiguities identified: The exact mathematical formula of the VioGén algorithm remains undisclosed by the Ministry of Interior.
• Alternative interpretations: The failures could be interpreted purely as human/organizational failures of the police force rather than algorithmic failures, though the systemic bias and resource-dependent thresholds point directly to algorithmic design issues.

The VioGen algorithmic risk assessment system in Spain failed to protect domestic violence victims due to systemic underestimation of risk and demographic biases. While causing tragic physical harm and violating fundamental human rights on a large scale, the national security threat remains limited to domestic societal stability and government police function efficacy.

• Overall national security impact: Substantial
• Scope: Single nation
• Primary target: Spain
• Alleged perpetrator: Unknown

Threat characteristics

• Imminence: Long-term. The issue represents an ongoing, systemic operational and strategic concern within the national police force rather than an immediate crisis.
• Autonomy: Human-supervised. The system acts as a risk assessor with human officers theoretically holding final decision-making power, though high automation bias (95% adherence) effectively reduces human oversight.
• Novelty: Established threat. Algorithmic bias and automation bias in public sector risk assessment tools are well-documented and established challenges.

Impact by dimension

• Physical security: Minor. While the system's failure indirectly led to tragic physical harm and homicides due to inadequate protection, it does not constitute a kinetic attack, critical infrastructure failure, or mass casualty threat to national security.
• Information security: Negligible. No information warfare, espionage, or intelligence security compromises were reported in connection with this system.
• Sovereignty: Minor. The incident involves domestic law enforcement decision-making processes, but does not threaten state authority, territorial control, or constitutional processes.
• Economic security: Negligible. No significant implications for national economic stability, strategic industries, or technological competitive advantage.
• Societal stability: Substantial. The system exhibits systematic demographic bias and fails to protect fundamental human rights, specifically the right to life and equal protection, affecting a large population of over 600,000 registered cases.