The first new study, published by BMJ<\/a> (affiliated with the British Medical Association) in a peer reviewed journal, and entitled \u201cGenerative artificial intelligence-driven chatbots and medical misinformation: an accuracy, referencing and readability audit<\/a>\u201d, studied five popular chatbots (Gemini, DeepSeek, Meta AI, ChatGPT and Grok), about one year ago, prompting each with 10 questions about things ranging from cancer to vaccines and nutrition, in open-ended dialogues, and reporting that nearly half of the responses were highly problematic. Worse, \u201cchatbot outputs were consistently expressed with confidence and certainty\u201d. The responses were also filled with hallucinations and fabricated citations. <\/p>\n All of this \u2013 the hallucinations, mistakes, and overconfidence \u2013 is entirely typical of LLMs, and entirely problematic in medicine. As the authors put it, in somewhat academic language, but entirely accurately, \u201ccontinued deployment without public education and oversight risks amplifying misinformation.\u201d<\/p>\n The second new study, published in JAMA Network Open, affiliated with the American Medical Association, called \u201cLarge Language Model Performance and Clinical Reasoning Tasks<\/a>\u201d looked at 21 frontier models across 29 questions, and reported that \u201cdespite progress, current LLMs remain limited in early diagnostic reasoning and cannot yet be relied on for unsupervised patient-facing clinical decision-making.\u201d<\/p>\n And the Post article actually only reported part of the new scientific literature on LLMs and medicines. Two other new studies that they missed only add to the concerns.<\/p>\n One, published in Nature Medicine, was called \u201cReliability of LLMs as medical assistants for the general public: a randomized preregistered study<\/a>\u201d. This one focused on \u201cwhether LLMs can assist members of the public in identifying underlying conditions and choosing a course of action\u201d. Again the results were both clear and troubling. LLMs \u201cidentified relevant conditions in fewer than 34.5% of cases\u2026 no better than [a] control group\u201d. Here the problem wasn\u2019t so much that the LLMs lacked access to proper information \u2014 the same study showed that the models could do better in the hands of trained physicians \u2014 but that patients don\u2019t know how to guide the LLMs to the right places. <\/p>\n In a recurring theme, we see that LLMs don\u2019t know what they don\u2019t know; they work decently well with the information they\u2019ve got but don\u2019t know how to conduct clinical interviews, and in the hands of the lay public can easily give bad advice because the proper questions never get asked, either by the patient or the LLMs. (An expert doctor might use the LLM to better effect, by asking the right questions.)<\/p>\n Still another new study, also published recently in Nature Medicine, entitled ChatGPT Health performance in a structured test of triage recommendations<\/a>, found that \u201cAmong gold-standard emergencies, the system undertriaged 52% of cases\u201d and concluded that \u201cThese findings reveal missed high-risk emergencies and inconsistent activation of crisis safeguards, raising safety concerns that warrant prospective validation before consumer-scale deployment of artificial intelligence triage systems.\u201d<\/p>\n As a scientist, I am always looking for converging evidence. Four studies in four journals published in the space of a few months reaching essentially the same conclusion is a crystal clear indicator that chatbots, especially when used by amateurs, simply cannot be trusted.<\/p>\n On a personal note, my friend Ben Riley lost his father recently, and Teddy Rosenbluth of The New York Times wrote a long, moving article about how his father<\/a> was mislead by A.I regarding his leukemia.<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/il\/wp-content\/uploads\/2026\/04\/https:\/\/substack-post-media.s3.amazonaws.com\/public\/images\/5a5a379c-b687-46fa-a180-95b24496aa5b_2420.jpeg\")
<\/a><\/p>\n