Australian Terrorism Prediction Tool Disparately Impacts Persons with Autism

An independent report found that the Vera-2R tool, designed to predict the risk of future terrorist activities, considered autism as a risk factor despite lacking empirical evidence to support this claim. The report called into question the tool's overall validity and reliability, stating it was "extremely poor" at accurately predicting risk. The inclusion of autism as a risk factor had potentially serious implications for the tool's use and credibility.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.3 Unequal performance across groups

The VERA2R tool performed unequally across demographic groups by systematically overestimating the risk of offending for individuals with autism spectrum disorders without any empirical basis.

Additional risk subdomains

• 7.3 Lack of capability or robustness: The independent report found the tools had a weak empirical basis and were extremely poor at predicting risk overall, representing a major capability failure.
• 5.1 Overreliance and unsafe use: Governments continued to rely on the tool to justify ongoing detention despite knowing it lacked empirical validity.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The risk stems from the VERA2R and Radar algorithmic tools producing unreliable and biased risk assessments post-deployment, which was an unintentional failure of the system's predictive capabilities.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used in law enforcement and justice contexts to assess recidivism and justify ongoing detention. This directly aligns with the EU AI Act's classification of high-risk AI systems used in law enforcement and administration of justice.

AI system and alleged parties

• AI system: Radar, Vera-2R
• AI purpose: Recidivism Prediction; Threat Detection
• Behaviour type: Tool
• Alleged developer: Unspecified
• Alleged deployer: New South Wales Government, Australian Federal Government
• Alleged harmed parties: people with autism, lawyers and other experts who were not informed of the tool's limitations, Individuals assessed as high-risk based on the flawed criteria, General public

Harm severity

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Minor, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes differential treatment of individuals with autism spectrum disorders.

Directly caused: The VERA2R tool considered individuals at greater risk of offending if they were autistic, despite having no empirical basis to do so.

Indirectly caused: Autistic offenders faced a higher likelihood of being classified as high-risk, leading to harsher post-sentence conditions compared to non-autistic offenders.

Inferred additional harm: It is likely that multiple autistic individuals in the Australian correctional system were unfairly subjected to stricter supervision or ongoing detention due to this systemic bias.

Civil rights

Reported: The report explicitly describes the tool being used to justify harsh post-sentence orders, including ongoing detention, which impacts civil liberties.

Directly caused: The tool was used to justify ongoing detention and harsh post-sentence orders for offenders, directly restricting their liberty based on invalid risk predictions.

Indirectly caused: The federal government withheld the critical report from offenders' lawyers, experts, and state partners, undermining the right to a fair legal process.

Inferred additional harm: Dozens of individuals may have had their civil rights and freedom of movement severely restricted based on unscientific and biased risk assessments.

Epistemic

Reported: The report describes the tool's developers claiming strong reliability and validity when almost 60% of the cited evidence base was non-empirical and less than half of the cited works accurately reflected the source texts.

Directly caused: The tool's documentation presented false or misleading claims about its scientific validity and empirical backing to users and governments.

Indirectly caused: N/A

Inferred additional harm: The propagation of unscientific risk assessment methodologies may have corrupted the broader legal and forensic psychology fields' understanding of terrorism risk factors.

People affected

• Occurrences reported: 1
• People reportedly harmed: 14
• People reportedly exposed: 14

Potential causes

Management

• Nondisclosure of Critical Report: The government withheld the critical report from lawyers and experts.
• Continued Use Despite Warnings: Governments continued using the tool after receiving the report.
• Lack of Transparency: The government hid the report under national security claims.

Technology

• Flawed Predictive Algorithm: The tool relied on unverified risk factors to predict recidivism.
• Poor Prediction Accuracy: The instrument was extremely poor at predicting actual risk.

Data Inputs

• Unscientific Risk Factors: Autism and non-compliance were included without empirical evidence.
• Non-Empirical Cited Evidence: Almost 60 percent of the cited evidence base was not empirical.
• Inaccurate Citations: Less than half of cited works accurately reflected source texts.

Human Factors

• Practitioner Over-reliance: Practitioners relied on the tool despite its weak evidence base.
• Developer Overclaim: Developers made unsupported claims of strong reliability.

Process and Methods

• Lack of Empirical Evaluation: The tool was deployed without thorough empirical evaluation.
• Flawed Factor Development: The process for developing risk factors lacked scientific rigor.

Regulatory Environment

• Lack of Independent Oversight: No regulatory body evaluated the tool before its adoption.

Information quality

• Classification confidence: High
• Reason for confidence: The report provides clear, detailed information about the academic review of the VERA2R and Radar tools, their specific flaws (such as the unscientific inclusion of autism), and how they were used by the Australian government. The facts are well-documented through freedom of information releases.
• Ambiguities identified: The exact number of individuals who actually faced ongoing detention solely due to the VERA2R tool is not fully specified, as it is described as one of several tools used.

The Australian government's multi-year use of the unvalidated VERA2R and Radar tools to justify post-sentence detention compromised counter-terrorism decision-making and infringed on civil liberties, particularly for autistic individuals. While presenting minor national security risks, it highlights significant governance and human rights concerns in state-deployed AI.

• Overall national security impact: Minor
• Response level: Moderate
• Scope: Single nation
• Primary target: Australia
• Alleged perpetrator: Australian Department of Home Affairs

Threat characteristics

• Imminence: Long-term. The incident represents a long-term systemic issue with algorithmic tool validation rather than an active, immediate crisis.
• Autonomy: Human-controlled. The AI tool served as a decision-support instrument, with human officials ultimately making the final decisions on detention.
• Novelty: Established threat. The use of unvalidated or biased risk-prediction algorithms in law enforcement and corrections is an established issue globally.

Impact by dimension

• Physical security: Negligible. No physical systems, critical infrastructure, or human safety threats were impacted by the failure of the risk assessment tools.
• Information security: Negligible. There is no indication of information warfare, intelligence compromise, or foreign espionage associated with this incident.
• Sovereignty: Minor. Core government counter-terrorism decision-making was compromised by using an unvalidated risk tool, representing a minor impact on administrative integrity.
• Economic security: Negligible. The incident did not threaten Australia's economic stability, strategic industries, or technological competitive advantage.
• Societal stability: Minor. The tool's use led to systematic discrimination against individuals with autism and unjustified ongoing detention, impacting civil liberties.