The Guardian<\/a> reported that this study found AI often did not properly assess the urgency of medical care needed, and also failed to detect suicidal ideation.<\/p>\nIn just over half of the cases ChatGPT Health assessed, it did not advise someone to go to hospital when their case was an emergency, which study authors found gave people a false sense of security that could cost them their life.<\/p>\n
While increasing use of AI by patients before and after consultations was inevitable with the rise of the technology, Dr Ayre says people need to be aware that using AI in this way carries inherent risks.<\/p>\n
\u201cWe have lots of research coming through that shows, yes, a lot of the time tools like ChatGPT will be accurate, but it\u2019s not all the time. And it can have really serious consequences even if all of the information is there for it to make a correct decision,\u201d she says.<\/p>\n
\u201cIt doesn\u2019t necessarily always give the correct advice. And in fact you can give it the same information multiple times and sometimes it will be wrong and sometimes it will be correct.\u201d<\/p>\n
\u201cThere\u2019s something about these AI tools that\u2019s really empowering for people and I think we need to recognise that, but still understand that this is a new technology, people are learning how to use it, they\u2019re still learning about the limitations and how to use it safely.\u201d<\/p>\n
\u201cWhat we found was that quite a notable number of people were asking higher risk questions. So this could be things like asking what do my symptoms mean, asking about treatment advice, or interpreting test results.\u201d<\/p>\n
\u201cAll of those kinds of questions, there\u2019s a higher risk that comes with them because they typically would require clinical expertise to answer them safely.\u201d<\/p>\n
![\"ChatGPT](\"https:\/\/www.newsbeep.com\/au\/wp-content\/uploads\/2026\/03\/shutterstock_2725133373-1024x576.jpg\")
A study showed that ChatGPT Health frequently advised someone not to go to hospital when their case was an emergency (Nwz \/ Shutterstock).<\/p>\n
Dr ChatGPT in the consultation room<\/p>\n
Dr Shaddy Hanna works as a general practitioner in Greater Western Sydney, as well as researching the impact of AI and digital scribes.<\/p>\n
He says he\u2019s noticed increasing numbers of people coming to their consultation having looked at ChatGPT first, and it\u2019s happening faster than many clinicians realise.<\/p>\n
\u201cWe used to talk about Dr Google, now it\u2019s Dr ChatGPT,\u201d Dr Hanna says.<\/p>\n
He\u2019s fascinated by how quickly patient behaviour has shifted, and how that\u2019s changed the way he practices as well.<\/p>\n
\u201cOur patients are using these tools to understand their symptoms, interpret test results, explore treatment options or even just to make sense of a health concern late at night when they can\u2019t access their GP,\u201d he says.<\/p>\n
\u201cAlmost overnight patients began arriving to consultations more informed and with more specific questions about their symptoms, their medical history and treatment options.\u201d<\/p>\n
But while this shift has been helpful in many consultations by supporting more focussed conversations and better shared decision-making, one of his greatest concerns is that people don\u2019t understand AI\u2019s limitations.<\/p>\n
\u201cLike any probabilistic system, AI tools can sometimes produce very confident answers that may not actually be clinically appropriate or reliable \u2014 nor disclose this,\u201d he says.<\/p>\n
\u201cThese days I try to routinely ask patients during consultations whether they\u2019ve gone to Dr Google or Dr ChatGPT to ask any specific questions before they came into see me that day.\u201d<\/p>\n
\u201cI think that creating space for these conversations with patients is important, because it allows us as trained clinicians, to help explore, contextualise, and critically analyse the advice they have been given.\u201d<\/p>\n
\u201cEspecially when the advice given may often seem so coherent and confident, and can mislead patients to trust in the advice given, almost as a replacement for consultations with trained medical practitioners.\u201d<\/p>\n
He\u2019s found AI\u2019s at times misleading clinical advice can have serious consequences, as well as privacy considerations around data use.<\/p>\n
\u201cWhen patients share symptoms and test results of identifiable health information with an AI platform, it\u2019s not always clear where the data goes or how it\u2019s stored,\u201d he says.<\/p>\n
\u201cAs clinicians, we can take these opportunities to help our patients think critically about their data stewardship, before they place their data blindly into the trust of commercial AI platforms.\u201d<\/p>\n
Dr Ayre agrees that stronger governance over AI platforms is key, with the TGA having an important role.<\/p>\n
\u201cWith the TGA, it also takes into account off label use. So if a manufacturer or a software developer becomes aware that people are using it outside of its intended use, they need to respond to that as well,\u201d she says.<\/p>\n
\u201cSo that could be either to disable the ability to complete those actions that would make it a medical device or to undergo that regulatory approach so that it can be classed as a medical device.\u201d<\/p>\n
Nance Haxton was a journalist at the ABC for nearly 20 years. She\u2019s also worked as an Advocate at the Disability Royal Commission helping people with disabilities tell their stories\u00a0and as a senior reporter for the National Indigenous Radio Service.\u00a0<\/p>\n
In that time she\u2019s won a\u00a0range of Australian and international honours, including two Walkley Awards, and three New York Festivals Radio Awards trophies.<\/p>\n
Now freelancing as The Wandering Journo, Nance is independently producing podcasts including her personal audio slice of Australia \u201cStreets of Your Town\u201d.<\/p>\n
Subscribe to the free\u00a0InSight+\u00a0weekly newsletter\u00a0here<\/a>. It is available to all readers, not just registered medical practitioners.<\/p>\n","protected":false},"excerpt":{"rendered":"Concerns are rising over patients\u2019 use of AI tools such as ChatGPT for health information, but with little…\n","protected":false},"author":2,"featured_media":545082,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":["post-545081","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-au","tag-australia","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/545081","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=545081"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/545081\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/545082"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=545081"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=545081"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=545081"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}