Image by Getty \/ Futurism<\/p>\n
Health practitioners are becoming increasingly uneasy about the medical community making widespread use of error-prone generative AI tools.<\/p>\n
The proliferation of the tech has repeatedly been hampered by rampant “hallucinations<\/a>,” a euphemistic term for the bots’ made-up facts and convincingly-told lies.<\/p>\n One glaring error proved so persuasive that\u00a0it took over a year to be caught. In their May 2024 research paper<\/a> introducing a healthcare AI model, dubbed Med-Gemini, Google researchers showed off the AI analyzing brain scans from the radiology lab for various conditions.<\/p>\n It identified an “old left basilar ganglia infarct,” referring to a purported part of the brain \u2014 “basilar ganglia” \u2014 that simply doesn’t exist in the human body. Board-certified neurologist Bryan Moore flagged the issue to The Verge<\/a>, highlighting that Google fixed its blog post about the AI \u2014 but failed to revise the research paper itself.<\/p>\n The AI likely conflated the basal ganglia, an area of the brain that’s associated with motor movements and habit formation, and the basilar artery, a major blood vessel at the base of the brainstem. Google blamed the incident on a simple misspelling of “basal ganglia.”<\/p>\n It’s an embarrassing reveal that underlines persistent and impactful shortcomings of the tech. Even the latest “reasoning” AIs by the likes of Google and OpenAI are spreading falsehoods<\/a> dreamed up by large language models that are trained on vast swathes of the internet.<\/p>\n In Google’s search results, this can lead to headaches<\/a> for users during their research and fact-checking efforts.<\/p>\n But in a hospital setting, those kinds of slip-ups could have devastating consequences. While Google’s faux pas more than likely didn’t result in any danger to human patients, it sets a worrying precedent, experts argue.<\/p>\n “What you\u2019re talking about is super dangerous,” healthcare system Providence’s chief medical information officer Maulin Shah told The Verge. “Two letters, but it\u2019s a big deal.”<\/p>\n Google touted its healthcare AI as having a “substantial potential in medicine” last year, arguing it could be used to identify conditions in X-rays, CT scans, and more.<\/p>\n After Moore flagged the mistake in the company’s research paper to Google, employees told him it was a typo. In its updated blog post<\/a>, Google noted that “‘basilar’ is a common mis-transcription of ‘basal’ that Med-Gemini has learned from the training data, though the meaning of the report is unchanged.”<\/p>\n Yet the research paper still erroneously refers to the “basilar ganglia” at the time of writing.<\/p>\n In a medical context, AI hallucinations could easily lead to confusion and potentially even put lives at risk.<\/p>\n “The problem with these typos or other hallucinations is I don\u2019t trust our humans to review them, or certainly not at every level,” Shah told The Verge.<\/p>\n It’s not just Med-Gemini. Google’s more advanced healthcare model, dubbed MedGemma, also led to varying answers depending on the way questions were phrased, leading to errors some of the time.<\/p>\n “Their nature is that [they] tend to make up things, and it doesn\u2019t say \u2018I don\u2019t know,\u2019 which is a big, big problem for high-stakes domains like medicine,” Judy Gichoya, Emory University associate professor of radiology and informatics, told The Verge.<\/p>\n Other experts say we’re rushing into adapting AI in clinical settings \u2014 from AI therapists<\/a>, radiologists<\/a>, and nurses<\/a> to patient interaction transcription services<\/a>\u00a0\u2014 warranting a far more careful approach.<\/p>\n In the meantime, it will be up to humans to continuously monitor the outputs of hallucinating AIs, which could counteractively lead to inefficiencies.<\/p>\n And Google is going full steam ahead. In March, Google revealed<\/a> that its extremely error-prone AI Overviews search feature would start giving health advice. It also introduced an “AI co-scientist” who would assist human scientists in discovering new drugs, among other “superpowers<\/a>.”<\/p>\n But if their outputs go unobserved and unverified, human lives could be at stake.<\/p>\n “In my mind, AI has to have a way higher bar of error than a human,” Shah told The Verge. “Maybe other people are like, \u2018If we can get as high as a human, we\u2019re good enough.\u2019 I don\u2019t buy that for a second.”<\/p>\n