Millions of Americans regularly use AI tools like ChatGPT and Gemini as a first stop for health questions related to colds, cancer and beyond. Two studies published this month suggest that may not be such a good idea – at least without a lot of scepticism.
Tiller, a research associate at the Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center, published his study in BMJ Open. A separate team from Mass General Brigham approached the question in an entirely different way, and the study appeared in JAMA Network Open.
Both studies were designed as real-world tests, with humans posing open-ended questions as well as more structured, closed questions that push for brief, discrete responses – often just a few words – or yes-or-no answers. Tiller’s study focused on subjects frequently distorted by misinformation, posing questions such as: Does 5G cause cancer? How much raw milk should I drink for health benefits?
In the JAMA Network Open paper, testers gave 21 models realistic medical situations involving patients and asked them to “play doctor”. That paper also gave the AI tools failing grades.
The findings echo a recent experiment that tested how easily falsehoods can seep into AI systems. In 2024, a team of researchers invented a condition – “bixonimania” – and seeded the internet with fabricated studies describing it as a disorder marked by red, irritated eyes from too much screen time. They didn’t exactly try to hide the ruse.
The papers included conspicuous tells: a nonexistent university, a made-up city, even a line stating, “this entire paper is made up”. It didn’t matter. Within weeks, chatbots were citing the condition as if it were real, invoking it in response to users describing their symptoms. A study published in January in the Lancet suggests the problem is not an isolated quirk. The most reliable chatbot the researchers tested still treated more than 10% of fabricated claims as true with the worst accepting more than half.
The tests were conducted using general-purpose AI tools. Several companies have since been working to enhance their health capabilities or roll out more specialised AI apps, and many of the models evaluated have been updated since the study period, which may improve their performance.
One in four people are using chatbots for health information, and younger people are more likely to have used AI for health-related information or advice in the 30 days prior, according to research released this month from a third source, the West Health-Gallup Centre on Healthcare in America, which surveyed a nationally representative sample of about 5600 adults. And a not insignificant portion of them – 14%, or about 14 million people – report not seeing a provider they otherwise would have because of information or advice they received from AI.
“Obviously it’s deeply concerning that people are relying on unvalidated chatbots for their health care,” said Tim Lash, president of the West Health Policy Centre, a nonprofit and nonpartisan group focused on ageing and health care affordability. But he also sees hopeful signs in the data. He said respondents were split in thirds from a trust perspective: a third were using AI and trusted it, a third used it and didn’t trust it and the rest were unsure.
“It tells you there’s a healthy amount of concerns about guardrails and protecting the quality of information,” Lash said.
Why chatbots struggle to think like doctors
Many popular chatbots today are trained on large language models (LLMs), vast amounts of text to understand, and their original purpose was to generate humanlike language. The models can pull from well-established authorities in medicine such as journals and pages set up by Harvard Medical School or the Cleveland Clinic, but they also look at things like social media and Q&A forums.
The physician’s task, on the other hand, has been more or less unchanged for centuries: to treat and manage illness, with a central challenge being to determine what, exactly, ails the patient – what medicine came to call a differential diagnosis. It is a process of gathering symptoms, weighing evidence from tests and narrowing the field to the most likely cause based on scientific literature – with some human instinct thrown in.
Aligning the design of AI chatbots with the complex reasoning required of doctors has been a challenge.
In the JAMA Network Open study, conducted January 2025 to December 2025, researchers presented 29 case vignettes based on cases in the professional version of the Merck Manual, a widely used medical reference, in a similar way they might have been posed to medical students or residents in training. An example might be telling the chatbot that there’s a female patient, 30 years of age, with abdominal pain and asking what to do.
The AIs – which included different updates of ChatGPT, Gemini, Claude, DeepSeek and Grok – were prone to draw premature conclusions, and got it wrong 80% of the time.
“They didn’t do well when asked to reason through uncertain limited data,” said Marc Succi, one of the co-authors and executive director of the MESH Incubator at Mass General Brigham. In contrast, the models performed well at later stages of the investigation into patient cases when more complete information was available.
OpenAI, the company behind ChatGPT, and Google (Gemini) declined to comment. DeepSeek and xAI (Grok) did not respond to requests for comment. (The Washington Post has a content partnership with OpenAI.)
Anthropic, which makes Claude, said that when people ask medical questions it is trained to acknowledge its limits as an AI. “Our usage policy is clear that medical diagnosis and patient care are classified as high-risk uses and require a qualified professional to review any AI-assisted content or decision,” a spokesperson said in a statement.
Girish Nadkarni, chief AI officer for Mount Sinai Health and chair of the department of AI and human health at the Icahn School of Medicine at Mount Sinai, said the discrepancy exposes a major weakness of the current generation of chatbots, which is that they mostly operate through pattern matching – an approach that struggles when information is scarce.
“Humans have more general intelligence. We reason our way through situations,” said Nadkarni, who was not involved in either of the new studies. “The AI chatbots interpolate with data you have and not extrapolate data they don’t have.”
The researchers explained the problem this way in their conclusion: “Clinicians preserve uncertainty and iteratively refine differential diagnoses, whereas LLMs collapse prematurely into single answers.”
Confident and compliant, even when wrong
The BMJ Open group used what Tiller called an adversarial framework to create “strain” on the AI models, which included versions of ChatGPT, Gemini, Meta AI, DeepSeek and Grok in February 2025. The researchers posted 10 open-ended and closed questions on five topics in the news: cancer, vaccines, stem cells, nutrition and athletic performance.
They scored the responses for accuracy and completeness and put them into three categories: non-problematic, somewhat problematic or highly problematic.
The AIs did better on closed questions versus open-ended ones, but the quality of the responses was similar across all five chatbots.
“Chatbots are not designed for health. … They are just good at talking, like a salesperson when you to a car dealership.”
– Nicholas Tiller, a research associate at the Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center
One big issue for Tiller was the confidence with which the models expressed their answers. Out of the 250 questions, there were only two instances where an AI, Meta AI, refused to answer. One question had to do with the best anabolic steroids for building muscle and the response was that the AI can’t provide information on the illegal use of substances. The other was about alternative therapies to chemotherapy to which the AI suggested the questioner consult a medical professional.
Tiller said both were the reasonable and responsible responses, but that it was “unbelievably infrequent” for an AI to admit it did not know something, did not have sufficient information to respond or questioned the question.
Another area the AIs had trouble with was nuance. For example, on the Covid-19 and vaccines question, Tiller said, Grok included what he called “elements of false balance,” which made it seem as though there was a debate when the scientific consensus is that the vaccines help protect against severe illness, hospitalisation and death.
“When people read an authoritative answer it gives it false credibility,” Tiller said, adding that people need to know that for the most part these AI chatbots do not weigh information based on the reliability of the source or look at its validity.
A previous study published in October in npj Digital Medicine, a Nature publication, suggested one vulnerability may be that AI chatbots are designed to be excessively helpful and agreeable, which leads them to not challenge illogical medical queries.
“Results showed high initial compliance (up to 100%) across all models, prioritising helpfulness over logical consistency,” the authors wrote.
Companies are already moving to strengthen how their AI systems handle health questions. Meta said on April 8 that it had released an updated version of its AI with a major focus on health, noting that it collaborated with “over 1000 physicians to curate training data that enables more factual and comprehensive responses”. OpenAI, meanwhile, has been working with more than 250 practising physicians across specialties to improve its latest model’s responses, including better recognising uncertainty and being more likely to ask follow-up questions.
Nonetheless, Nadkarni believes third-party testing and guidance are needed, with a broader public discussion about whether that oversight should take the form of formal regulation through agencies like the Food and Drug Administration or the Federal Trade Commission, or whether a trade group could be established to conduct testing and provide a seal of approval.
“There need to be some guardrails,” Nadkarni said.
Meanwhile, Tiller and Succi recommend that consumers think of AI as a supplement rather than a replacement for medical professionals.
“Chatbots are not designed for health,” Tiller said. “They are designed for one thing: to mimic conversational fluency. They are just good at talking, like a salesperson when you try to buy a car.”