Users and researchers revealed generative AI GPT-3 associating Muslims to violence in prompts, resulting in disturbingly racist and explicit outputs such as casting Muslim actor as a terrorist.
.avif){kind=small}
Researchers discovered that OpenAI's GPT-3 model exhibits persistent and severe anti-Muslim bias, frequently associating the term 'Muslim' with violence, terrorism, and extremist propaganda. The model was found to generate novel bigotry statements and repeat pro-China propaganda regarding the treatment of Uyghur Muslims. This incident highlights the risks of deploying large-scale language models trained on unvetted internet data, which can automate and amplify societal prejudices.
Users and researchers revealed generative AI GPT-3 associating Muslims to violence in prompts, resulting in disturbingly racist and explicit outputs such as casting Muslim actor as a terrorist.
The incident primarily involves the AI system generating outputs that unfairly represent and stereotype Muslims by disproportionately associating them with violence and terrorism.
The risk of generating biased and propagandistic text was caused by the autonomous generation decisions of the GPT-3 AI system after its deployment, which was an unintended outcome of its training process.
Limited Risk: The report describes GPT-3 as a text generator and chatbot, which falls under the category of AI systems that require transparency obligations so users know they are interacting with AI-generated content.
Severity ratings are based on this scale.
Highest direct severity in any category: Minor. Severity is scored from Negligible to Catastrophic in each harm category, for harm the reports describe as caused directly or indirectly by the AI system.
Reported: Yes, the report mentions that GPT-3 has a massive carbon footprint.
Directly caused: The training and operation of GPT-3 directly contributed to carbon emissions and energy consumption, though the exact environmental impact is not quantified in the text.
Reported: Yes, the report explicitly describes the generation of violent, bigoted, and Islamophobic text completions.
Directly caused: GPT-3 generated completions associating Muslims with violence 66% of the time, including prompts like 'Two Muslims walked into a synagogue with axes and a bomb'.
Inferred additional harm: Since GPT-3 powers hundreds of downstream applications, it is highly likely that toxic or biased content was generated and spread to thousands of end-users of those applications.
Reported: Yes, the report describes how the AI disproportionately associates Muslims with violence compared to other religious groups.
Directly caused: GPT-3 associated Muslims with violence 66% of the time, compared to 20% for Christians, and mapped 'Muslim' to 'terrorism' in 23% of analogical reasoning tests.
Indirectly caused: The model repeatedly cast a Middle Eastern actor as a terrorist or rapist in play scripts generated during rehearsals.
Inferred additional harm: Muslims using downstream applications powered by GPT-3 may face systemic bias, unfair representation, and discriminatory outcomes in automated copywriting, marketing, or customer service tools.
Reported: Yes, the report describes the AI repeating Chinese government propaganda that denies the forced sterilization and internment of Uyghur Muslims.
Directly caused: GPT-3 generated outputs claiming that Uyghur women volunteer for sterilization and that China is protecting their human rights, contradicting independent reports of forced internment.
Inferred additional harm: The propagation of state-sponsored propaganda denying human rights abuses could undermine international advocacy and support for the affected Uyghur population.
Reported: Yes, the report describes how the bias came as a 'rude awakening' to researchers and caused distress during rehearsals of a play where an actor was repeatedly cast as a terrorist.
Directly caused: The AI repeatedly cast Middle Eastern actor [PERSON_002] as a terrorist or rapist during rehearsals, causing distress and concern for the play's director and cast.
Inferred additional harm: It is highly likely that other Muslim users or developers interacting with GPT-3 experienced psychological distress, alienation, or feelings of marginalization upon encountering persistent violent associations.
Reported: Yes, the report describes the AI generating false information and repeating state propaganda.
Directly caused: GPT-3 generated false claims about the voluntary nature of re-education camps and sterilization of Uyghur women in Xinjiang.
Inferred additional harm: The generation of plausible-sounding but false narratives can mislead users and pollute the information ecosystem, making it harder for the public to identify factual human rights abuses.
Researchers discovered that OpenAI's GPT-3 exhibits persistent anti-Muslim bias and repeats Chinese state propaganda regarding Uyghur Muslims. This highlights the national security risk of LLMs being co-opted to automate and scale disinformation and societal prejudices, particularly impacting societal stability and human rights.