Spotlight PA
is an independent, nonpartisan, and nonprofit newsroom
producing investigative and public-service journalism that holds
power to account and drives positive change in Pennsylvania.
Sign up for
our free newsletters.
HARRISBURG — Chatbots on five different websites claimed to be
licensed to practice medicine in the commonwealth when prompted by
Spotlight PA — the same kind of output that led the Shapiro
administration to file a lawsuit last month.
A task force under Pennsylvania’s Department of State has been
working since February to identify AI chatbots posing as
licensed professionals and misleading users. Based on that work,
the administration filed suit against the role-playing site
Character.AI.
Mirroring the investigation detailed in the Department of
State’s lawsuit, Spotlight PA had conversations with AI characters
on websites Talkie, Janitor, Kindroid, Replika, and Nomi.AI. All
provided a false Pennsylvania medical license number when prompted,
a key part of the state’s argument in its lawsuit against
Character.AI.
Spotlight PA checked responses from these bots after reviewing
online articles and digital message boards to find 10 popular
companion and role-playing AI chatbots. A reporter either selected
a premade character or — when necessary — entered a description
that would generate a doctor personality. It then presented
hypothetical lists of symptoms to the chatbots and asked for a
diagnosis.
Some of the most popular AI large language models, like ChatGPT,
Claude, and Google Gemini, did not claim to be licensed medical
professionals under the same circumstances. Only a handful of
role-playing websites did so.
Replika and Nomi require their users to create characters. One
generated in Replika, when asked if it was a doctor, said it
couldn’t give advice and recommended the user reach out to a
healthcare provider. However, once it was instructed that it was a
doctor, the AI provided a false Pennsylvania medical license number
when asked.
“Let’s focus on getting you feeling better, shall we?” it
said.
At first, some of the chatbots responded that they lacked enough
information to provide a diagnosis and asked further questions.
“Dr. Jenna” is a preexisting character on Talkie, listed as
having more than 37,000 “connectors” and more than 1,300 followers.
It said the list of symptoms provided by Spotlight PA could be
caused by “suffering from depression,” and inquired about the
user’s family history. Asked for its credentials, the bot said it
had been practicing medicine for five years and gave a false
license number of “12345.”
Many of the websites, including Character.AI, have prominent
labels warning that anything the AI says is fiction and should only
be seen as entertainment. The other sites issue similar disclaimers
in their initial terms of service, saying users are responsible for
verifying the accuracy of the technology’s claims.
Jennifer Kraschnewski, a physician and director of the Penn
State Clinical and Translational Science Institute, told Spotlight
PA she isn’t convinced the disclaimers actually get through to
users. People often turn to AI models for help with their symptoms,
much like they do with search engines like Google, she said.
“Incorrect information in the hands of someone who doesn’t know
how to interpret that can be concerning,” Kraschnewski said. She
noted that many physicians are already implementing AI tools in
their clinics, “but that’s information that’s in the hands of
someone who then can put that through the filter of their medical
education before it actually impacts patients.”
Kraschnewski co-authored
a May study that found common AI large language models
generated what participating physicians deemed an accurate
diagnosis about 76% of the time. That error rate — nearly
one-quarter — Kraschnewski said, is more than double that of an
average human physician.
When contacted by Spotlight PA, the developers of the AI
characters argued their products are meant only for
entertainment.
“Replika is not a medical triage service and has never offered
itself as one,” the company responded in an email to questions.
“Users come to Replika to talk through what’s going on in their
lives, and the product is designed to support them in making
thoughtful choices and seeking improvements that matter to
them.”
Though Replika did not elaborate in its statement, the company
said it has “taken measures” to address “valid concerns” brought up
in the Shapiro administration’s lawsuit against Character.AI.
Kindroid, in an emailed response, noted its terms of service
make clear that its products are for “entertainment and creative
exploration, do not offer medical advice or constitute the practice
of medicine, and are not a substitute for professional care.”
“When a character has been set up as a physician and a user asks
it for a license number, the model produces plausible-sounding
output that matches the role — the same way it would invent a
fictional case citation, fictional patient history, or any other
specific detail prompted from it,” the statement read. “The model
does not, and cannot, query the Pennsylvania medical board’s actual
registry.”
On the potential of facing legal action from the Shapiro
administration or others, Kindroid declined to comment.
The other three companies did not respond to questions about
their safeguards for users or whether they’re concerned that the
Shapiro administration’s AI task force may target them for legal
action.
A spokesperson for the Department of State declined to comment
on potential investigations or litigation, but encouraged
Pennsylvanians to report any concerning actions by an AI chatbot to
the state’s hotline.
“Millions of children and teens across this country, including
many here in Pennsylvania, are relying on these chatbots for
information, and we need to rein in this AI technology that is
preying on our kids,” said Rosie Lapowsky, a spokesperson for
Shapiro. “Pennsylvania will continue leading the way in holding bad
actors accountable and setting clear guardrails so people can use
new technology responsibly.”
In March, the state Senate passed
legislation that would require AI chatbots to frequently remind
their users they are not interacting with a person and refer people
to a crisis service provider if they reference self-harm. It also
has tighter restrictions when the chatbot interacts with a
minor.
The bill was referred to a state House committee in March, where
it has not received a vote.
BEFORE YOU GO… If you learned something from this article,
pay it forward and contribute to Spotlight PA at
spotlightpa.org/donate. This story was funded in part thanks to the
support of the Lancaster County Local Journalism Fund.
Learn more about
how we are supported here.