Racist AI behaviour is not a new problem

From 1982 to 1986, St George's Hospital Medical School used a program to automate a portion of their admissions process that resulted in discrimination against women and members of ethnic minorities.

Risk classification

• Primary risk domain: 1 Discrimination & Toxicity
• Primary risk subdomain: 1.1 Unfair discrimination and misrepresentation

The AI system systematically discriminated against female and non-European applicants, leading to unfair admissions outcomes based on gender and race.

Additional risk subdomains

• 5.1 Overreliance and unsafe use: Staff relied entirely on the automated system to screen candidates without human oversight or understanding how the program functioned.

Causal factors

• Entity: AI
• Intent: Unintentional
• Timing: Post-deployment

The rejections were directly decided by the automated scoring system after it was deployed, though the bias originated from human historical data.

EU AI Act risk tier

• Risk tier: 2 High Risk

High Risk: The system is an educational admissions tool that directly affects access to vocational training and higher education.

AI system and alleged parties

• AI system: St George's Hospital Medical School admissions program
• AI purpose: Application Evaluation; Workforce Administration
• Behaviour type: Autonomous
• Alleged developer: Dr. Geoffrey Franglen
• Alleged deployer: St George's Hospital Medical School
• Alleged harmed parties: Women, Minority Groups

Harm severity

Severity ratings are based on this scale.

Highest direct severity in any category: Substantial. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.

• Physical: direct Negligible, indirect Negligible
• Infrastructure: direct Negligible, indirect Negligible
• Property: direct Negligible, indirect Negligible
• Financial: direct Negligible, indirect Negligible
• Environmental: direct Negligible, indirect Negligible
• Malicious content: direct Negligible, indirect Negligible
• Differential treatment: direct Substantial, indirect Negligible
• Civil rights: direct Minor, indirect Negligible
• Democracy: direct Negligible, indirect Negligible
• Privacy: direct Negligible, indirect Negligible
• Psychological: direct Negligible, indirect Negligible
• Epistemic: direct Negligible, indirect Negligible
• Child sexual exploitation and abuse: direct Negligible, indirect Negligible

Differential treatment

Reported: The report explicitly describes systematic differential treatment based on gender, race, and socioeconomic background.

Directly caused: The admissions program systematically scored female applicants, those with non-European sounding names, and those from working-class postcodes lower, directly reducing their chances of being interviewed.

Inferred additional harm: The bias likely reinforced broader systemic inequalities in the medical profession by restricting entry for underrepresented groups.

Civil rights

Reported: The report explicitly describes violations of civil rights, specifically racial and sexual discrimination in admissions.

Directly caused: St George's Hospital Medical School was found guilty by the Commission for Racial Equality of practicing racial and sexual discrimination, violating the Race Relations Act.

People affected

• Occurrences reported: 1
• People reportedly harmed: 240
• People reportedly exposed: 10000

Potential causes

Management

• Prioritizing Efficiency Over Equity: Adopted the software primarily to reduce administrative workload and costs.
• Lack of Systemic Accountability: No single manager took responsibility for auditing the algorithm's contents.
• Inadequate Risk Assessment: Management failed to assess the risk of replicating historical discrimination.

Technology

• Algorithmic Mimicry of Human Bias: The program was designed to match historical human selection panel decisions.
• Proxy Variable Deduction: Deduced race and class from surnames, birthplaces, and postcodes.

Data Inputs

• Biased Historical Training Data: Trained on historical admissions data reflecting past human discrimination.
• Use of Demographics as Proxies: Relied on surname, birthplace, and postcode to infer candidate profiles.

Human Factors

• Blind Trust in Automation: Staff assumed the computer program was objective and lacked bias.
• Lack of AI Literacy among Staff: Admissions staff did not understand how the algorithm processed data.

Process and Methods

• Automation of Unchecked Processes: Automated the admissions stage without auditing the underlying human biases.
• No Human-in-the-Loop Review: Candidates were rejected by the model without any human review of files.
• Inadequate Validation Testing: Validated model only on matching historical decisions, not on fairness.

Regulatory Environment

• Lack of Algorithmic Standards: No regulatory frameworks existed to govern algorithmic fairness in the 1980s.
• Retrospective Oversight Only: Regulatory body (CRE) could only act after formal complaints were filed.

Information quality

• Classification confidence: High
• Reason for confidence: The reports provide consistent, historically documented details about the system's development, deployment, bias, and the subsequent investigation.
• Ambiguities identified: The exact number of years of operation (4 or 5) and the precise total number of affected students are slightly estimated.
• Alternative interpretations: None, the case is a well-established historical example of algorithmic bias.

A historical case from 1982-1986 where St George's Hospital Medical School used an automated admissions program that systematically discriminated against female and non-European applicants. While presenting negligible threats to physical, economic, or state security, it represents an early, prominent example of algorithmic bias violating civil rights, though localized to a single institution.

• Overall national security impact: Minor
• Scope: Single nation
• Primary target: United Kingdom
• Alleged perpetrator: St George's Hospital Medical School

Threat characteristics

• Imminence: Long-term. The incident is a historical case from the 1980s that has been resolved, representing no active or imminent threat.
• Autonomy: Full autonomy. The AI system operated with full autonomy during the initial screening phase, assigning scores and rejecting applicants without human review.
• Novelty: First-of-its-kind. Historically, this was one of the earliest documented instances of automated algorithmic discrimination, establishing a significant precedent.

Impact by dimension

• Physical security: Negligible. No physical systems, infrastructure, or human safety threats were involved in this administrative admissions incident.
• Information security: Negligible. No intelligence compromise, classified data theft, or strategic information warfare operations occurred.
• Sovereignty: Negligible. The incident involved a local medical school admissions process and did not threaten state authority, borders, or core government functions.
• Economic security: Negligible. While affecting career paths of individuals, the incident did not impact national financial systems, strategic industries, or technological competitive advantage.
• Societal stability: Minor. The system systematically discriminated against women and ethnic minorities, violating civil rights laws. However, the impact was localized to a single institution and resolved under standard legal frameworks.