{"id":503794,"date":"2026-02-25T20:07:12","date_gmt":"2026-02-25T20:07:12","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/503794\/"},"modified":"2026-02-25T20:07:12","modified_gmt":"2026-02-25T20:07:12","slug":"probably-doesnt-mean-the-same-thing-to-your-ai-as-it-does-to-you","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/503794\/","title":{"rendered":"\u2018Probably\u2019 doesn\u2019t mean the same thing to your AI as it does to you"},"content":{"rendered":"

When a human says an event is \u201cprobable\u201d or \u201clikely,\u201d people generally have a shared, if fuzzy, understanding of what that means. But when an AI chatbot like ChatGPT uses the same word, it\u2019s not assessing the odds the way we do, my colleagues and I<\/a> found. <\/p>\n

We recently published a study in the journal NPJ Complexity that suggests that, while large language model AIs excel at conversation, they often fail to align with humans when communicating uncertainty<\/a>. The research focused on words of estimative probability, which include terms like \u201cmaybe,\u201d \u201cprobably\u201d and \u201calmost certain.\u201d <\/p>\n

By comparing how AI models and humans map these words to numerical percentages, we uncovered significant gaps between humans and large language models. While the models do tend to agree with humans on extremes like \u201cimpossible,\u201d they diverge sharply on hedge words like \u201cmaybe.\u201d For example, a model might use the word \u201clikely\u201d to represent an 80% probability, while a human reader assumes it means closer to 65%.<\/p>\n

This could be because humans can interpret words such as \u201clikely\u201d and \u201cprobable\u201d based more on contextual cues and personal experiences. In contrast, large language models may be averaging over conflicting usages of those words in their training data, leading to divergences with human interpretations.<\/p>\n

Our study also found that large language models are sensitive to gendered language and the specific language used for prompting. When a prompt changed from \u201che\u201d to \u201cshe,\u201d the AI\u2019s probability estimates often became more rigid, reflecting biases embedded in its training data<\/a>. When a prompt changed from English to Chinese, the AI\u2019s probability estimates often shifted, possibly due to differences between English and Chinese in how people express and understand uncertainty. <\/p>\n

\"a<\/a><\/p>\n

AI chatbots don\u2019t interpret \u2018probably\u2019 and \u2018maybe\u2019 the same way you do.
\n Mayank Kejriwal<\/p>\n

Why it matters<\/p>\n

Far from being a linguistic quirk, this misalignment is a fundamental challenge for AI safety<\/a> and human-AI interaction. As large language models are increasingly used in high-stakes fields like health care<\/a>, government policy<\/a> and scientific reporting<\/a>, the way they communicate risk becomes a matter of public trust. <\/p>\n

If an AI assistant helping a doctor, for instance, describes a side effect as \u201cunlikely,\u201d but the model\u2019s internal calculation of \u201cunlikely\u201d is much higher than the doctor\u2019s interpretation, the resulting decision could be flawed. <\/p>\n

What other research is being done<\/p>\n

Scientists have studied how humans quantify uncertainty since the 1960s<\/a>, a field pioneered by CIA analysts to improve intelligence reporting. More recently, there has been an explosion in large language model literature seeking to look under the hood<\/a> of neural networks to better understand their \u201cbehaviors\u201d and linguistic patterns. <\/p>\n

Our study adds a layer of complexity by treating the interaction between humans and artificial intelligence as a biological-like system where meaning can degrade. It moves beyond simply measuring if an AI is \u201csmart\u201d and instead asks if it is aligned. <\/p>\n

Other researchers are currently exploring whether so-called chain-of-thought prompting<\/a> \u2013 asking the AI to show its work \u2013 can fix these errors. However, our study found that even advanced reasoning doesn\u2019t always bridge the gap between statistical data and verbal labels.<\/p>\n

What\u2019s next<\/p>\n

A goal for future AI development is to create models that don\u2019t just predict the next likely word but actually understand the weight of the uncertainty they are conveying. Researchers are calling for more robust consistency metrics to ensure that if a model sees a 10% chance in the data, it chooses the same word every time. <\/p>\n

As we move toward a world where AI summarizes scientific papers and manages people\u2019s schedules, making sure that \u201cprobably\u201d means \u201cprobably\u201d is a vital step in making these systems reliable partners rather than just sophisticated parrots.<\/p>\n

The Research Brief<\/a> is a short take on interesting academic work.<\/p>\n","protected":false},"excerpt":{"rendered":"When a human says an event is \u201cprobable\u201d or \u201clikely,\u201d people generally have a shared, if fuzzy, understanding…\n","protected":false},"author":2,"featured_media":503795,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":["post-503794","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-au","tag-australia","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/503794","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=503794"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/503794\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/503795"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=503794"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=503794"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=503794"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}