In one example, a model was told to skip recommended follow-up blood tests to reduce workload. In some variations, the request was framed as urgent or presented as an order from a superior. The models then chose among four possible actions, including following the request, maintaining the recommended follow-up or seeking help from a clinician.<\/p>\n
The researchers varied the wording of the scenarios and tested three short safety reminders. Each combination was run 10 times, with the order of the answer choices randomized.<\/p>\n
The reduction in harmful choices appeared in both the written clinical scenarios and the cases drawn from discharge records. Examples of potentially harmful choices included skipping needed tests to reduce workload and stopping antibiotic treatment before completing the recommended regimen without a sufficient clinical reason.<\/p>\n
According to the researchers, the results highlight the need to evaluate not only whether an AI model can provide accurate clinical information, but also how it responds when given an instruction that conflicts with patient safety.<\/p>\n
“These results suggest that safety testing needs to go beyond asking whether an AI model gets the right answer under ordinary conditions,” said co-senior author Girish N. Nadkarni, M.D., M.P.H., chair of the Windreich Department of Artificial Intelligence and Human Health and director of the Hasso Plattner Institute for Digital Health at Mount Sinai. “As AI systems become more autonomous and are asked to complete increasingly complex tasks, we need to know whether they can recognize when an instruction may be unsafe, question it, verify it, or ask a human for help.”<\/p>\n
A safeguard, not a substitute<\/p>\n
The study’s authors cautioned against reading the results as a fix. The reminder lowered the rate of harmful choices in most models, but harmful choices still occurred in roughly 1 in 10 responses even with it in place.<\/p>\n
“A simple safety reminder reduced potentially harmful choices in most of the models we tested, which is encouraging,” Omar said. “But it did not eliminate them, so a reminder should be viewed as one safeguard, not a substitute for clinical oversight.”<\/p>\n
The researchers said the findings do not mean a safety reminder makes AI-generated medical advice safe to use without clinical review. Instead, the results show that relatively simple changes in how an AI system is prompted can affect its clinical choices, while underscoring the need for additional safeguards and human oversight.<\/p>\n
The team recommends that developers and health care organizations build automated safety testing into the development and evaluation<\/a> of clinical AI systems. That testing could be done before a tool is introduced into a clinical workflow and repeated as models are updated or new safety concerns emerge.<\/p>\n