{"id":95071,"date":"2025-08-26T23:30:21","date_gmt":"2025-08-26T23:30:21","guid":{"rendered":"https:\/\/www.newsbeep.com\/uk\/95071\/"},"modified":"2025-08-26T23:30:21","modified_gmt":"2025-08-26T23:30:21","slug":"something-extremely-scary-happens-when-advanced-ai-tries-to-give-medical-advice-to-real-world-patients","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/uk\/95071\/","title":{"rendered":"Something Extremely Scary Happens When Advanced AI Tries to Give Medical Advice to Real World Patients"},"content":{"rendered":"

Image by Getty \/ Futurism<\/p>\n

Last week, Google AI pioneer Jad Tarifi sparked controversy when he told Business Insider<\/a> that it no longer makes sense<\/a> to get a medical degree \u2014 since, in his telling, artificial intelligence will render such an education obsolete by the time you’re a practicing doctor.<\/p>\n

Companies have long touted the tech as a way to free up the time<\/a> of overworked doctors and even aid them in specialized skills, including scanning medical imagery for tumors<\/a>. Hospitals have already been rolling out AI tech to help with administrative work<\/a>.<\/p>\n

But given the current state of AI\u00a0\u2014 from widespread hallucinations<\/a> to “deskilling<\/a>” experienced by doctors over-relying on it \u2014 there’s reason to believe that med students should stick it out.<\/p>\n

If anything, in fact, the latest research suggests we need human healthcare professionals now more than ever.<\/p>\n

As PsyPost reports<\/a>, researchers have found that frontier AI models fail spectacularly\u00a0when the familiar formats of medical exams are even\u00a0slightly altered, greatly undermining their ability to help patients in the real world \u2014 and raising the possibility that, instead, they could cause great harm by providing garbled medical advice in high-stakes health scenarios.<\/p>\n

As detailed in a paper<\/a> published in the journal JAMA Network Open, things quickly fell apart for models including OpenAI’s GPT-4o and Anthropic’s Claude 3.5 Sonnet when the wording of questions in a benchmark test was only slightly adjusted.<\/p>\n

The idea was to probe the nature of how large language models arrive at their answers: by predicting the probability of each subsequent word \u2014 and not through a human-level understanding of complex medical terms.<\/p>\n

“We have AI models achieving near perfect accuracy on benchmarks like multiple-choice based medical licensing exam questions,” Stanford University PhD student and coauthor Suhana Bedi told PsyPost. “But this doesn\u2019t reflect the reality of clinical practice. We found that less than five percent of papers evaluate LLMs on real patient data, which can be messy and fragmented.”<\/p>\n

The results left a lot to be desired. According to Bedi, “most models (including reasoning models) struggled” when it came to “Administrative and Clinical Decision Support tasks.”<\/p>\n

The researchers suggest that “complex reasoning scenarios” in their benchmark threw the AIs for a loop since they “couldn\u2019t be solved through pattern matching alone” \u2014 which happens to be “exactly the kind of clinical thinking that matters in real practice,” per Bedi.<\/p>\n

“With everyone talking about deploying AI in hospitals, we thought this was a very important question to answer,” Bedi told PsyPost.<\/p>\n

For their benchmark test, the researchers made a clever adjustment to trip up the AIs: they replaced the correct answers of multiple-choice questions with the option “none of the other answers.” This change forced the AI models to actually reason their way to the right answer \u2014 and not rely on picking up familiar language patterns.<\/p>\n

The team noticed a significant decline in accuracy when presented with their new test, as compared to their answers to the original questions. For instance, OpenAI’s GPT-4o showed a reduction of 25 percent, while Meta’s Llama model showed a drop of almost 40 percent.<\/p>\n

The results suggest current AI systems may be vastly over-relying on recognizing language patterns, making them inadequate for real-world clinical use.<\/p>\n

“It\u2019s like having a student who aces practice tests but fails when the questions are worded differently,” Bedi told PsyPost. “For now, AI should help doctors, not replace them.”<\/p>\n

The research highlights the importance of finding new ways to evaluate the proficiency of AI models. That’s especially true for an extremely high-stakes environment like a hospital.<\/p>\n

“Until these systems maintain performance with novel scenarios, clinical applications should be limited to nonautonomous supportive roles with human oversight,” the researchers wrote in their paper.<\/p>\n

More on AI doctors: Founder of Google’s Generative AI Team Says Don’t Even Bother Getting a Law or Medical Degree, Because AI’s Going to Destroy Both Those Careers Before You Can Even Graduate<\/a><\/p>\n

<\/p>\n","protected":false},"excerpt":{"rendered":"Image by Getty \/ Futurism Last week, Google AI pioneer Jad Tarifi sparked controversy when he told Business…\n","protected":false},"author":2,"featured_media":95072,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[43],"tags":[102,2960,56,54,55],"class_list":{"0":"post-95071","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-healthcare","8":"tag-health","9":"tag-healthcare","10":"tag-uk","11":"tag-united-kingdom","12":"tag-unitedkingdom"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/95071","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=95071"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/95071\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/95072"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=95071"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=95071"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=95071"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}