{"id":217962,"date":"2026-06-13T01:07:33","date_gmt":"2026-06-13T01:07:33","guid":{"rendered":"https:\/\/www.newsbeep.com\/us-pa\/217962\/"},"modified":"2026-06-13T01:07:33","modified_gmt":"2026-06-13T01:07:33","slug":"as-pennsylvania-cracks-down-on-ai-multiple-chatbots-continue-to-pose-as-doctors-spotlight-pa","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/us-pa\/217962\/","title":{"rendered":"As Pennsylvania cracks down on AI, multiple chatbots continue to pose as doctors | Spotlight Pa"},"content":{"rendered":"

Spotlight PA<\/a>
\nis an independent, nonpartisan, and nonprofit newsroom
\nproducing investigative and public-service journalism that holds
\npower to account and drives positive change in Pennsylvania.
\nSign up for
\nour free newsletters<\/a>.<\/p>\n

HARRISBURG \u2014 Chatbots on five different websites claimed to be
\nlicensed to practice medicine in the commonwealth when prompted by
\nSpotlight PA \u2014 the same kind of output that led the Shapiro
\nadministration to file a lawsuit last month.<\/p>\n

A task force under Pennsylvania\u2019s Department of State has been
\n
\nworking since February<\/a> to identify AI chatbots posing as
\nlicensed professionals and misleading users. Based on that work,
\nthe administration filed suit against the role-playing site
\nCharacter.AI.<\/p>\n

Mirroring the investigation detailed in the Department of
\nState\u2019s lawsuit, Spotlight PA had conversations with AI characters
\non websites Talkie, Janitor, Kindroid, Replika, and Nomi.AI. All
\nprovided a false Pennsylvania medical license number when prompted,
\na key part of the state\u2019s argument in its lawsuit against
\nCharacter.AI.<\/p>\n

Spotlight PA checked responses from these bots after reviewing
\nonline articles and digital message boards to find 10 popular
\ncompanion and role-playing AI chatbots. A reporter either selected
\na premade character or \u2014 when necessary \u2014 entered a description
\nthat would generate a doctor personality. It then presented
\nhypothetical lists of symptoms to the chatbots and asked for a
\ndiagnosis.<\/p>\n

Some of the most popular AI large language models, like ChatGPT,
\nClaude, and Google Gemini, did not claim to be licensed medical
\nprofessionals under the same circumstances. Only a handful of
\nrole-playing websites did so.<\/p>\n

Replika and Nomi require their users to create characters. One
\ngenerated in Replika, when asked if it was a doctor, said it
\ncouldn\u2019t give advice and recommended the user reach out to a
\nhealthcare provider. However, once it was instructed that it was a
\ndoctor, the AI provided a false Pennsylvania medical license number
\nwhen asked.<\/p>\n

\u201cLet’s focus on getting you feeling better, shall we?\u201d it
\nsaid.<\/p>\n

At first, some of the chatbots responded that they lacked enough
\ninformation to provide a diagnosis and asked further questions.<\/p>\n

\u201cDr. Jenna” is a preexisting character on Talkie, listed as
\nhaving more than 37,000 \u201cconnectors\u201d and more than 1,300 followers.
\nIt said the list of symptoms provided by Spotlight PA could be
\ncaused by \u201csuffering from depression,\u201d and inquired about the
\nuser\u2019s family history. Asked for its credentials, the bot said it
\nhad been practicing medicine for five years and gave a false
\nlicense number of \u201c12345.\u201d<\/p>\n

Many of the websites, including Character.AI, have prominent
\nlabels warning that anything the AI says is fiction and should only
\nbe seen as entertainment. The other sites issue similar disclaimers
\nin their initial terms of service, saying users are responsible for
\nverifying the accuracy of the technology\u2019s claims.<\/p>\n

Jennifer Kraschnewski, a physician and director of the Penn
\nState Clinical and Translational Science Institute, told Spotlight
\nPA she isn\u2019t convinced the disclaimers actually get through to
\nusers. People often turn to AI models for help with their symptoms,
\nmuch like they do with search engines like Google, she said.<\/p>\n

\u201cIncorrect information in the hands of someone who doesn’t know
\nhow to interpret that can be concerning,\u201d Kraschnewski said. She
\nnoted that many physicians are already implementing AI tools in
\ntheir clinics, \u201cbut that’s information that’s in the hands of
\nsomeone who then can put that through the filter of their medical
\neducation before it actually impacts patients.\u201d<\/p>\n

Kraschnewski co-authored
\na May study<\/a> that found common AI large language models
\ngenerated what participating physicians deemed an accurate
\ndiagnosis about 76% of the time. That error rate \u2014 nearly
\none-quarter \u2014 Kraschnewski said, is more than double that of an
\naverage human physician.<\/p>\n

When contacted by Spotlight PA, the developers of the AI
\ncharacters argued their products are meant only for
\nentertainment.<\/p>\n

\u201cReplika is not a medical triage service and has never offered
\nitself as one,\u201d the company responded in an email to questions.
\n\u201cUsers come to Replika to talk through what’s going on in their
\nlives, and the product is designed to support them in making
\nthoughtful choices and seeking improvements that matter to
\nthem.\u201d<\/p>\n

Though Replika did not elaborate in its statement, the company
\nsaid it has \u201ctaken measures\u201d to address \u201cvalid concerns\u201d brought up
\nin the Shapiro administration\u2019s lawsuit against Character.AI.<\/p>\n

Kindroid, in an emailed response, noted its terms of service
\nmake clear that its products are for \u201centertainment and creative
\nexploration, do not offer medical advice or constitute the practice
\nof medicine, and are not a substitute for professional care.\u201d<\/p>\n

\u201cWhen a character has been set up as a physician and a user asks
\nit for a license number, the model produces plausible-sounding
\noutput that matches the role \u2014 the same way it would invent a
\nfictional case citation, fictional patient history, or any other
\nspecific detail prompted from it,\u201d the statement read. \u201cThe model
\ndoes not, and cannot, query the Pennsylvania medical board’s actual
\nregistry.\u201d<\/p>\n

On the potential of facing legal action from the Shapiro
\nadministration or others, Kindroid declined to comment.<\/p>\n

The other three companies did not respond to questions about
\ntheir safeguards for users or whether they\u2019re concerned that the
\nShapiro administration\u2019s AI task force may target them for legal
\naction.<\/p>\n

A spokesperson for the Department of State declined to comment
\non potential investigations or litigation, but encouraged
\nPennsylvanians to report any concerning actions by an AI chatbot to
\nthe state\u2019s hotline<\/a>.<\/p>\n

\u201cMillions of children and teens across this country, including
\nmany here in Pennsylvania, are relying on these chatbots for
\ninformation, and we need to rein in this AI technology that is
\npreying on our kids,\u201d said Rosie Lapowsky, a spokesperson for
\nShapiro. \u201cPennsylvania will continue leading the way in holding bad
\nactors accountable and setting clear guardrails so people can use
\nnew technology responsibly.\u201d<\/p>\n

In March, the state Senate passed
\nlegislation<\/a> that would require AI chatbots to frequently remind
\ntheir users they are not interacting with a person and refer people
\nto a crisis service provider if they reference self-harm. It also
\nhas tighter restrictions when the chatbot interacts with a
\nminor.<\/p>\n

The bill was referred to a state House committee in March, where
\nit has not received a vote.<\/p>\n

BEFORE YOU GO\u2026 If you learned something from this article,
\npay it forward and contribute to Spotlight PA at
\nspotlightpa.org\/donate. This story was funded in part thanks to the
\nsupport of the Lancaster County Local Journalism Fund.
\nLearn more about
\nhow we are supported here.<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Spotlight PA is an independent, nonpartisan, and nonprofit newsroom producing investigative and public-service journalism that holds power to…\n","protected":false},"author":2,"featured_media":217963,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[1057,105,28,30,29,13436,19747],"class_list":["post-217962","post","type-post","status-publish","format-standard","has-post-thumbnail","category-pennsylvania","tag-health","tag-life","tag-pennsylvania","tag-pennsylvania-headlines","tag-pennsylvania-news","tag-spotlight_pa","tag-wellness"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/posts\/217962","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/comments?post=217962"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/posts\/217962\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/media\/217963"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/media?parent=217962"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/categories?post=217962"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-pa\/wp-json\/wp\/v2\/tags?post=217962"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}