UT Austin GRADE Algorithm Allegedly Reinforced Historical Inequalities

The University of Texas at Austin's Department of Computer Science's assistive algorithm to assess PhD applicants "GRADE" raised concerns among faculty about worsening historical inequalities for marginalized candidates, prompting its suspension.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The GRADE system was criticized for encoding and replicating historical biases, leading to potential unfair discrimination against applicants based on gender, race, and academic background.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Human reviewers' opinions were likely colored by the algorithm's low scores, creating an overreliance on the automated screening.
• 7.4 Lack of transparency or interpretability: The machine-learning algorithm did not provide clear guidance on how it classified or scored applicants, making its decisions difficult to interpret.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The potential bias and unfair screening of applicants were unintended consequences of deploying the GRADE algorithm to streamline the admissions process.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is used in education and vocational training to evaluate and screen applicants for admission, which directly affects access to education.

AI system and alleged parties

• AI system: None named
• AI purpose: Application Evaluation; Resource Allocation
• Behaviour type: Assistant
• Alleged developer: University of Texas at Austin researchers
• Alleged deployer: University of Texas at Austin's Department of Computer Science
• Alleged harmed parties: University of Texas at Austin PhD applicants of marginalized groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Minor, indirect Minor
• Civil rights: direct Minor, indirect Minor
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes potential differential treatment of applicants from underrepresented groups caused by the algorithm.

Directly caused: The GRADE algorithm scored applicants using features that correlate with demographic groups, potentially giving lower scores to women, minorities, and graduates of non-elite schools.

Indirectly caused: Applicants scored lower by the algorithm received less attention from human reviewers, as full reviews were reduced by 71 percent.

Inferred additional harm: It is highly likely that hundreds of qualified minority and female applicants were systematically disadvantaged or rejected due to the biased scoring system.

Civil rights

Reported: The report explicitly describes concerns regarding equity, diversity, and fairness in the admissions process, which relate to civil rights.

Directly caused: The system potentially violated equal opportunity principles by replicating historical biases in admissions.

Indirectly caused: The use of the algorithm may have contributed to the ongoing underrepresentation of women and minorities in the computer science field.

Inferred additional harm: Systemic exclusion of marginalized groups from advanced educational opportunities at a top-tier institution.

People affected

• Occurrences reported: 1
• People reportedly exposed: 6000

Potential causes

Management

• Prioritizing Speed over Equity: Management prioritized saving review time over ensuring fair admissions.
• Lack of Risk Assessment: Deployed the tool without assessing long-term impacts on student diversity.

Technology

• Algorithmic Bias Replication: Replicated and locked in historical biases of the 2013 admissions committee.
• Lack of Model Interpretability: ML algorithms do not give clear guidance on how they classify applicants.
• Software Maintenance Issues: Changes in data and software environments made the system hard to maintain.

Data Inputs

• Biased Training Data: System trained on historical data reflecting past inequities and human bias.
• Proxy Variables for Bias: Variables like elite university names and letter keywords encoded social bias.
• Unstructured Data Simplification: Turned letters of recommendation and personal statements into a bag of words.

Human Factors

• Automation Bias: Low scores from the algorithm colored human reviewers' opinions of applicants.
• Overreliance on Efficiency: Staff accepted reduced reviews for low-scored applicants to save time.
• Unconscious Bias in Scoring: Reviewers relied on subjective factors like letters containing specific words.

Process and Methods

• Lack of Algorithmic Auditing: No effort was made to audit the impact of scores across demographic groups.
• Inadequate Disclosure: Applicants were not told they were screened by an algorithm.
• Single Human Reviewer Safeguard: Using only one human reviewer instead of multiple passes reduced oversight.

Regulatory Environment

• Lack of External Standards: No clear regulations or standards governed AI use in university admissions.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide clear, consistent details about the GRADE algorithm's purpose, training data, deployment timeline, and the specific criticisms regarding bias. The university's response and the creators' perspectives are also well-documented.
• Ambiguities identified: The exact demographic impact of the algorithm was never formally audited, so the precise level of bias or number of affected applicants remains unquantified.
• Alternative interpretations: The university claimed the system was discontinued primarily due to maintenance difficulties, whereas critics attribute it entirely to bias and ethical concerns.

The University of Texas at Austin's use of the GRADE machine-learning tool from 2013 to 2019 highlighted concerns about algorithmic bias replicating historical human prejudices in admissions. While raising significant ethical and equity concerns regarding access to higher education, the incident remains a localized academic issue with negligible direct national security implications.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United States
• Alleged perpetrator: University of Texas at Austin

Threat characteristics

• Imminence: Long-term. The system was discontinued in 2020, making this a historical case study rather than an active or developing threat.
• Autonomy: Human-supervised. The AI system assisted in screening and prioritizing applicants to reduce review times, but human reviewers ultimately made the final admissions decisions.
• Novelty: Established threat. Algorithmic bias and the replication of historical human prejudices in training datasets are well-documented and established issues in machine learning.

Impact by dimension

• Physical security: Negligible. The tool was used strictly for academic graduate admissions screening and had no connection to physical systems, critical infrastructure, or kinetic operations.
• Information security: Negligible. The incident did not involve any intelligence operations, classified information compromise, or systematic information warfare campaigns.
• Sovereignty: Negligible. Academic admissions processes at a state university do not impact core sovereign government functions, territorial control, or electoral systems.
• Economic security: Negligible. While affecting computer science education, the incident did not threaten national economic stability, strategic industries, or technological competitive advantage.
• Societal stability: Minor. The system potentially encoded and reinforced systemic demographic biases in graduate admissions, representing a localized civil rights and fairness concern but not a large-scale threat to societal stability.