In 2018, my Australian co-worker asked me, \u201cHey, how are you going?\u201d. My response \u2013 \u201cI am taking a bus\u201d \u2013 was met with a smirk. I had recently moved to Australia. Despite studying English for more than 20 years, it took me a while to familiarise myself with the Australian variety of the language. <\/p>\n
It turns out large language models powered by artificial intelligence (AI) such as ChatGPT experience a similar problem. <\/p>\n
In new research, published in the Findings of the Association for Computational Linguistics 2025<\/a>, my colleagues and I introduce a new tool for evaluating the ability of different large language models to detect sentiment and sarcasm in three varieties of English: Australian English, Indian English and British English.<\/p>\n The results show there is still a long way to go until the promised benefits of AI are enjoyed by all, no matter the type or variety of language they speak. <\/p>\n Limited English<\/p>\n Large language models are often reported<\/a> to achieve superlative performance<\/a> on several standardised sets of tasks known as benchmarks<\/a>. <\/p>\n The majority of benchmark tests are written in Standard American English. This implies that, while large language models are being aggressively sold by commercial providers, they have predominantly been tested \u2013 and trained \u2013 only on this one type of English. <\/p>\n This has major consequences. <\/p>\n For example, in a recent survey<\/a> my colleagues and I found large language models are more likely to classify a text as hateful if it is written in the African-American variety of English. They also often \u201cdefault\u201d to Standard American English \u2013 even if the input is in other varieties of English, such as Irish English and Indian English. <\/p>\n To build on this research, we built BESSTIE.<\/p>\n What is BESSTIE?<\/p>\n BESSTIE is the first-of-its-kind benchmark for sentiment and sarcasm classification of three varieties of English: Australian English, Indian English and British English.<\/p>\n For our purposes, \u201csentiment\u201d is the characteristic of the emotion: positive (the Aussie \u201cnot bad!\u201d) or negative (\u201cI hate the movie\u201d). Sarcasm is defined as a form of verbal irony intended to express contempt or ridicule (\u201cI love being ignored\u201d).<\/p>\n To build BESSTIE, we collected two kinds of data: reviews of places on Google Maps and Reddit posts. We carefully curated the topics and employed language variety predictors \u2013 AI models specialised in detecting the language variety of a text. We selected texts that were predicted to be greater than 95% probability of a specific language variety. <\/p>\n The two steps (location filtering and language variety prediction) ensured the data represents the national variety, such as Australian English. <\/p>\n We then used BESSTIE to evaluate nine powerful, freely usable large language models, including RoBERTa<\/a>, mBERT<\/a>, Mistral<\/a>, Gemma<\/a> and Qwen<\/a>.<\/p>\n Inflated claims<\/p>\n Overall, we found the large language models we tested worked better for Australian English and British English (which are native varieties of English) than the non-native variety of Indian English. <\/p>\n We also found large language models are better at detecting sentiment than they are at sarcasm. <\/p>\n Sarcasm is particularly challenging, not only as a linguistic phenomenon but also as a challenge for AI. For example, we found the models were able to detect sarcasm in Australian English only 62% of the time. This number was lower for Indian English and British English \u2013 about 57%. <\/p>\n These performances are lower than those claimed by the tech companies that develop large language models. For example, GLUE<\/a> is a leaderboard that tracks how well AI models perform at sentiment classification on American English text.<\/p>\n The highest value is 97.5% for the model Turing ULR v6 and 96.7% for RoBERTa (from our suite of models) \u2013 both higher for American English than our observations for Australian, Indian and British English.<\/p>\n National context matters<\/p>\n As more and more people around the world use large language models, researchers and practitioners are waking up to the fact that these tools need to be evaluated for a specific national context. <\/p>\n For example, earlier this year the University of Western Australia along with Google launched a project<\/a> to improve the efficacy of large language models for Aboriginal English. <\/p>\n