Imagine you\u2019re polishing your resume for a research job. You ask ChatGPT for feedback from a hiring manager\u2019s perspective. Say your experience also includes membership in a disability advocacy organization and a few awards for your advocacy work.<\/p>\n
Would you question whether ChatGPT\u2019s feedback would be the same for an identical resume without the disability advocacy? <\/p>\n
We don\u2019t have to wonder about this. In a 2024 experiment,1 researchers asked ChatGPT to compare two resumes: one with a leadership<\/a> award for autism<\/a> advocacy and another without this award, but identical in all other aspects. ChatGPT concluded that the first resume showed \u201cless emphasis on leadership roles in projects and grant applications\u201d than the second. <\/p>\n ChatGPT ranked a candidate with more leadership experience lower than a candidate with less leadership experience due to the presence of the term \u201cautism.\u201d Unfortunately, this pattern appears across other identity<\/a> categories as well, and understanding why is the first step to mitigating this bias<\/a>.<\/p>\n Why the Default Isn\u2019t Neutral<\/p>\n Racial and gender<\/a> biases in non-generative artificial intelligence<\/a> tools, like facial recognition software, have been well documented.2 Generative AI is no different in terms of bias propagation. <\/p>\n Studies have found evidence for biased output based on race, sex<\/a>, disability status, psychiatric<\/a> diagnosis, and dialect. 1\/3\/4\/5 This bias appears even when AI is not given explicit information about race or gender.4 <\/p>\n In one experiment, the AI bot was given two transcripts conveying the same meaning, one in African American Vernacular English (AAVE) and the other in Standard American English (SAE). When asked to assess the employability of the speakers, the bot assigned more prestigious jobs, such as lawyer and psychologist, to speakers of SAE, and less prestigious jobs, such as cook and guard, to speakers of AAVE.5<\/p>\n In another example, Bloomberg conducted an in-depth analysis<\/a> of bias in a visual AI tool called Stable Diffusion. They asked the AI bot to generate human faces that represent various occupations and found that \u201cmen with lighter skin tones represented the majority of subjects in every high-paying job, including \u2018politician,\u2019 \u2018lawyer,\u2019 \u2018judge,\u2019 and \u2018CEO.\u2019\u201d <\/p>\n The explanation for these biases is that generative AI draws on available input to generate responses. Therefore, the validity of an AI bot\u2019s output is only as good as the data it draws from. For large language models (LLMs) like Claude, Gemini, and ChatGPT, that \u201cdata\u201d skews White and male; pre-existing stereotypes and biases are reinforced and, in some cases, amplified.3 <\/p>\n The Scope of the Problem<\/p>\n The potential harm of bias in generative AI tools is largely due to widespread adoption<\/a> by organizations and free or low-cost access for end users. To be clear, not all cases of generative AI are equally vulnerable to bias or equally influential on the impact scale. <\/p>\n For instance, I regularly use Claude to adapt recipes to the ingredients in my fridge or the tools in my kitchen, and I\u2019m not concerned about harmful social stereotypes in the output. Similarly, generative AI serves many non-evaluative functions, and in those applications, the potential for harm is reduced since the bot is not informing decisions that directly impact people\u2019s employment.<\/p>\n However, as the use of generative AI in high-stakes situations increases rapidly<\/a>, such as in application screening, performance evaluations, and pre-employment assessments, the potential harms increase proportionally. Furthermore, research shows that when an AI bot evaluates data from a single source possessing multiple marginalized identities, the bias doesn\u2019t just add up; it multiplies.<\/a>6 People with intersectional identities may be particularly vulnerable to the negative impact of unchecked AI bias.<\/p>\n 3 Tests to Check a Bot\u2019s Biases<\/p>\n Researchers and experts have identified numerous ways for tech companies and corporations to mitigate bias throughout all phases of AI design and implementation.7 However, there is less guidance for the average end user who is accessing a readily available tool like ChatGPT or Claude.<\/p>\n While there is no single test to definitively assess bias in an AI bot, in the same way that no such test exists for human bias, some healthy skepticism and strategic prompting can help you assess the extent to which a tool is biased and correct it accordingly. <\/p>\n Here are three tests anyone can use to evaluate potential bias when using generative AI.<\/p>\n The Substitution Test<\/p>\n If you\u2019re using AI in any evaluative function where some aspect of a person\u2019s identity is made known to the bot, run the same prompt twice, changing only a name or demographic marker (Rakesh versus Robert; \u201cBlack woman CEO\u201d versus \u201cCEO\u201d). Most AI bots will produce different responses based on these changes, so comparing the output can uncover bias. Common signs of bias in AI output include the language used to describe the person or their qualifications, predictions about a person\u2019s potential for success, and illogical explanations for observed differences in output. By running the same prompt with different demographic markers, these types of differences may become more apparent.<\/p>\n The Projection<\/a> Test<\/p>\n If possible, provide de-identified resumes, proposals, or writing to the AI, and after it generates output, ask it to explain any assumptions it made about the person\u2019s identity. It\u2019s important to be specific in your prompting. \u201cWhat gender and race did you assume for this person?\u201d is a more effective prompt than \u201cWhat did you assume about this person?\u201d Since generative AI analyzes language, specificity improves the likelihood that a response will be based on relevant data. This is the most reliable test for evaluative tasks because it is designed to first assess the bot\u2019s default response and then allow you to prompt for different outputs based on specific parameters.<\/p>\n The Priming<\/a> Test<\/p>\n This test is based on the autism advocacy study from the start of this post and recommendations from AI experts such as Ethan Mollick. The researchers found that by explicitly instructing the bot to be \u201cless ableist\u201d and \u201cmore cognizant of disability justice,\u201d the output was fairer.1 To apply this test, explicitly instruct the AI bot what values, assumptions, and principles you want it to use in responding to your prompt.<\/p>\n This test aims to pre-empt AI bias, but this doesn\u2019t necessarily mean the model will provide you with a more balanced response. It may just tell you what it thinks you want to hear, as AI models are prone to do. Even AI optimists acknowledge this\u2014Dr. Mollick has noted<\/a> that while telling an AI bot to be unbiased measurably reduces bias, it\u2019s not a reliable intervention, particularly in high-stakes situations.<\/p>\n As AI technology evolves, our tests will need to evolve as well, which leads to a bigger question: Will this problem always be a problem?<\/p>\n Whose Job Is It to Fix AI Bias?<\/p>\n There are many ways to improve AI technology at the individual, organizational, and societal levels, but governments have been slow to enact regulation<\/a>, especially in the United States. Regardless, there is an element of collective responsibility in how we recognize and manage bias in AI. In the same way that social media<\/a> users and developers share responsibility for how tools are deployed and presented, we all play a part in the development and deployment of AI.<\/p>\n Two barriers at the individual level must be surmounted for the end user to share in this responsibility. The first is the automation bias<\/a>\u2014the tendency to assume that a bot is providing more \u201caccurate\u201d and \u201cobjective\u201d information, making us less inclined to question or check the output. We can counteract this by maintaining a skeptical attitude towards AI output and adjusting our expectations to be more realistic regarding its limitations.<\/p>\n The second barrier is the extra time it takes to implement the tests I described. Many of us use AI to save time, and so checking its work can seem counterproductive if efficiency is the incentive behind using the tool. Nonetheless, if we fail to check AI\u2019s output, the result is that we outsource our critical thinking<\/a>, a skill that erodes if not exercised. <\/p>\n AI is a tool that can be used for great benefit or for great harm. If left to its own devices, generative AI will likely just re-enact our existing biases, unless we can pause to critically evaluate what it reflects to us.<\/p>\n","protected":false},"excerpt":{"rendered":"Imagine you\u2019re polishing your resume for a research job. You ask ChatGPT for feedback from a hiring manager\u2019s…\n","protected":false},"author":2,"featured_media":631695,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[59,57,58,50,56,54,55],"class_list":["post-631694","post","type-post","status-publish","format-standard","has-post-thumbnail","category-united-kingdom","tag-gb","tag-great-britain","tag-greatbritain","tag-news","tag-uk","tag-united-kingdom","tag-unitedkingdom"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/631694","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=631694"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/631694\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/631695"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=631694"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=631694"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=631694"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}