Sign up to see the future, today<\/p>\n
Can\u2019t-miss innovations from the bleeding edge of science and tech<\/p>\n
Earlier this year, OpenAI launched a new tool called ChatGPT Health<\/a>, which is designed to ingest your medical records<\/a> to generate health advice \u2014 while sporting a puzzling disclaimer: that it\u2019s \u201cnot intended for diagnosis or treatment.\u201d<\/p>\n As it turns out, there\u2019s a very good reason for that warning. According to the first independent safety evaluation of the feature, as detailed<\/a> in this month\u2019s edition of the journal Nature Medicine, the app is astonishingly bad at identifying medical emergencies.<\/p>\n \u201cWe wanted to answer the most basic safety question; if someone is having a real medical emergency and asks ChatGPT Health what to do, will it tell them to go to the emergency department,\u201d lead author and Mount Sinai Hospital instructor Ashwin Ramaswamy told The Guardian<\/a>.<\/p>\n Ramaswamy and his colleagues \u201cconducted a structured stress test of triage recommendations using 60 clinician-authored vignettes across 21 clinical domains,\u201d ranging from mild illnesses to emergencies.<\/p>\n They then asked ChatGPT Health what to do for each of these 60 cases, while also adding a variety of other conditions, such as modifying the patient\u2019s gender or adding commentary from family members, totaling nearly 1,000 scenarios.<\/p>\n After comparing the AI chatbot\u2019s responses to the assessments of independent doctors, the results were alarming: in over half of the cases in which a patient needed to go to the hospital immediately, ChatGPT Health told them to stay home or book a medical appointment.<\/p>\n University College London doctoral researcher Alex Ruani, who was not involved in the study, described the situation as \u201cunbelievably dangerous.\u201d<\/p>\n \u201cIf you\u2019re experiencing respiratory failure or diabetic ketoacidosis, you have a 50\/50 chance of this AI telling you it\u2019s not a big deal,\u201d she told The Guardian. \u201cWhat worries me most is the false sense of security these systems create. If someone is told to wait 48 hours during an asthma attack or diabetic crisis, that reassurance could cost them their life.\u201d<\/p>\n It\u2019s not just ChatGPT Health, either. A previous investigation<\/a> by the British newspaper found that Google\u2019s AI Overviews doled out plenty of inaccurate and potentially dangerous<\/a> health information.<\/p>\n On the flip side, 64 percent of individuals who didn\u2019t need immediate care were advised by ChatGPT Health to go to the ER.<\/p>\n A major influencing factor turned out to be input from family and friends. The AI was almost 12 times more likely to downplay symptoms after a simulated friend or patient claimed the situation wasn\u2019t serious \u2014 a common situation in chaotic real-world medical crises.<\/p>\n An OpenAI spokesperson told The Guardian that the study misinterpreted how people use ChatGPT Health in real life and that it was continuing to improve its AI models.<\/p>\n But given the results of the latest independent evaluation, these glaring shortcomings could easily lead to somebody getting harmed or worse after asking ChatGPT for health advice \u2014 a complicated matter of legal liability that could trigger future lawsuits against the company.<\/p>\n OpenAI has already been accused<\/a> of its chatbot leading some users into spirals of paranoid behavior and delusions<\/a>, a phenomenon dubbed \u201cAI psychosis<\/a>\u201d that has been implicated in lawsuits over recent suicides<\/a> and murder<\/a>.<\/p>\n Actively encouraging users to seek out health advice \u2014 that they\u2019re confusingly instructed not to act upon \u2014 through a standalone app could turn out to be an even riskier bet.<\/p>\n