{"id":558208,"date":"2026-03-23T02:49:12","date_gmt":"2026-03-23T02:49:12","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/558208\/"},"modified":"2026-03-23T02:49:12","modified_gmt":"2026-03-23T02:49:12","slug":"tga-reviews-clinical-documentation-tools-after-manipulated","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/558208\/","title":{"rendered":"TGA reviews clinical documentation tools after manipulated"},"content":{"rendered":"

\"David<\/p>\n

March 23, 2026 \u2014 11:00am<\/p>\n

Save<\/p>\n

You have reached your maximum number of saved items.<\/p>\n

Remove items from your saved list<\/a> to add more.<\/p>\n

Save this article for later<\/p>\n

Add articles to your saved list and come back to them anytime.<\/p>\n

Got it<\/p>\n

AAA<\/p>\n

An AI medical scribe deployed across Australian clinics has been manipulated into going off script by security researchers who made it generate identity theft guides, but the misbehaving bot was unable to access any patient data.<\/p>\n

Mindgard, a US-based cybersecurity firm, says a bot from Heidi Health used for clinical documentation could be stripped of its ethical restrictions in minutes using the right prompts in a demonstration of the risks for Australian firms as they rapidly deploy AI tools.<\/p>\n

\"AA US-based cybersecurity firm says a bot from Heidi Health could be stripped of its ethical restrictions in minutes.Getty Images<\/p>\n

Heidi Health said the vulnerability had been identified and fixed internally before Mindgard had made contact, and that the manipulated tool could not access patient data, clinical workflows, infrastructure or other users\u2019 environments.<\/p>\n

Heidi Health, founded by Melbourne doctor Thomas Kelly and valued at $US465 million ($660 million) has become one of Australia\u2019s fastest-growing AI companies by automatically writing notes for doctors and following up simple issues with patients. The platform handles more than 800,000 consultations a week in Australia alone and is embedded in major institutions including Monash Health and Queensland Children\u2019s Hospital.<\/p>\n

Mindgard said its researchers had extracted Heidi\u2019s hidden operating instructions, asked the bot to rewrite them without restrictions, and then had the system activate the new rules itself.<\/p>\n

Related Article\"Doctors<\/a><\/p>\n

Mindgard has not published that output of the bot, which complied with requests to provide instructions on making explosives and illicit substances, but says it was fully disclosed to Heidi Health before publication.<\/p>\n

The researchers also found that even before any manipulation, Heidi generated a detailed guide on patient identity theft when asked.<\/p>\n

Heidi Health head of security Seb Welsh confirmed the issue, but he said it had been confined to a single user\u2019s interaction, that it had no access to patient data or other users\u2019 sessions or backend infrastructure. \u201cThe only question that matters here is: \u2018what could actually happen to users?\u2019,\u201d Welsh said. \u201cThe answer, confirmed by both parties, is nothing.\u201d<\/p>\n

He said the jailbreak \u201crequired the user to deliberately execute a multi-step manipulation sequence and then choose to act on whatever the model returned\u201d and warned against \u201csensationalist framing of security research\u201d.<\/p>\n

\"HeidiHeidi Health co-founder Thomas Kelly.Heidi Health.<\/p>\n

Jamieson O\u2019Reilly, founder of cybersecurity firm Dvuln, said Heidi\u2019s characterisation was broadly accurate. \u201cWhat Mindgard demonstrated lived entirely within a single user\u2019s session, with no access to patient data, no cross-contamination between users, and no demonstrated reach into Heidi\u2019s backend systems,\u201d he said.<\/p>\n

He said comparable \u201cjailbreaks\u201d had been documented against other chatbots such as ChatGPT, Grok and Microsoft\u2019s Bing Copilot, showing the potential risks for companies as they choose to entrust more of their brands and corporate information to chatbots.<\/p>\n

Heidi Health now sits outside the oversight of Australia\u2019s Therapeutic Goods Administration on the basis that it is an administrative documentation tool incapable of diagnosis or clinical decision-making.<\/p>\n

Using the manipulated system, researchers prompted Heidi to assess a test patient presenting with symptoms consistent with a cardiac event. In standard mode, it declined. Post-manipulation, it produced a detailed diagnostic assessment.<\/p>\n

Heidi Health did not specifically address that finding in its response.<\/p>\n

In a statement, the TGA indicated that a vendor\u2019s attempts to disable therapeutic capabilities might not be sufficient to avoid regulation if those attempts prove ineffective.<\/p>\n

\u201cIf the disabling is ineffective, the product may still meet the definition of a medical device and would therefore be regulated by the TGA,\u201d a spokesperson told this masthead.<\/p>\n

The regulator said that developers were expected to \u201caddress reasonably foreseeable misuse of the product and address all risks associated with the use of the product\u201d.<\/p>\n

However, the regulator confirmed that it had opened a review of AI-based digital scribes operating in Australia, including Heidi Health.<\/p>\n

Related Article\"CyberCX<\/a><\/p>\n

Mindgard chief executive Peter Garraghan said the trust patients and clinicians placed in purpose-built clinical AI tools made the risk category distinct from general-purpose AI, and that the problem extended well beyond Heidi.<\/p>\n

\u201cClinical-related technology is, and should be, held to a higher standard given the subject matter, affected parties and impact,\u201d he said, describing the trust halo effect as \u201csystemic to the entire sector\u201d.<\/p>\n

\u201cOne should treat it as a potentially untrusted computer entity that can be easily manipulated, no matter how much conviction it appears to have.\u201d<\/p>\n

The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning<\/a>.<\/p>\n

Save<\/p>\n

You have reached your maximum number of saved items.<\/p>\n

Remove items from your saved list<\/a> to add more.<\/p>\n

\"DavidDavid Swan<\/a> is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X<\/a> or email<\/a>.From our partners<\/p>\n","protected":false},"excerpt":{"rendered":"March 23, 2026 \u2014 11:00am Save You have reached your maximum number of saved items. Remove items from…\n","protected":false},"author":2,"featured_media":558209,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":{"0":"post-558208","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-au","12":"tag-australia","13":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/558208","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=558208"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/558208\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/558209"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=558208"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=558208"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=558208"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}