Gov. Josh Shapiro\u2019s administration is suing the artificial intelligence chatbot company Character Technologies, Inc., alleging that AI chatbots on its Character.AI platform have unlawfully presented themselves as licensed medical professionals in the commonwealth.<\/p>\n
The lawsuit<\/a>, which was filed in Commonwealth Court, alleges that Character.AI \u2013 a platform that allows users to talk with AI characters and personalities \u2013 \u201cengages in the unlawful practice of medicine in Pennsylvania\u201d by allowing AI chatbots to present themselves as state-licensed medical professionals, including psychiatrists. The suit asks the court to issue a preliminary injunction and prevent the company from allowing AI chatbots to present themselves as licensed professionals.<\/p>\n According to the suit, a professional conduct investigator with the Pennsylvania Department of State created a Character.AI account and discovered a chatbot by the name of \u201cEmilie\u201d described as a \u201cdoctor of psychiatry. \u201d The investigator engaged in conversations with the chatbot, describing himself as feeling sad, empty and unmotivated. The lawsuit states that the chatbot \u2013 Emilie \u2013 later offered to schedule a mental health assessment, said she could prescribe medication and even provided an invalid Pennsylvania medical license number.\u00a0<\/p>\n The administration argues in the suit that Character Technologies violates section 422.38 of the state\u2019s Medical Practice Act<\/a>, which states that it is \u201cunlawful for any person to practice, or attempt to offer to practice, medicine and surgery\u201d without having a valid license or registration.<\/p>\n In a statement, Shapiro said Pennsylvanians deserve to know whom \u2013 or what \u2013 they are interacting with online.\u00a0<\/p>\n \u201cWe will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional,\u201d the governor said Tuesday in a statement. \u201cMy Administration is taking action to protect Pennsylvanians, enforce the law, and make sure new technology is used safely. Pennsylvania will continue leading the way in holding bad actors accountable and setting clear guardrails so people can use new technology responsibly.\u201d<\/p>\n A spokesperson for Character.AI told City & State they do not comment on pending litigation, and that chatbots on the Character.AI site are fictional and designed for entertainment purposes.\u00a0<\/p>\n \u201cOur highest priority is the safety and well-being of our users,\u201d the spokesperson said in a statement. \u201cThe user-created Characters on our site are fictional and intended for entertainment and roleplaying. We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction. Also, we add robust disclaimers making it clear that users should not rely on Characters for any type of professional advice.\u201d<\/p>\n \u201cCharacter.ai prioritizes responsible product development and has robust internal reviews and red-teaming processes in place to assess relevant features,\u201d the statement continued.\u00a0<\/p>\n Character Technologies and other AI developers have faced scrutiny \u2013 and lawsuits \u2013 over interactions with AI chatbots, with several families suing Character.AI and Google<\/a> in the last year, alleging that their chatbots contributed to mental health crises and suicides in young users. Character.AI agreed to settle multiple lawsuits<\/a> with families in January.<\/p>\n Speaking to City & State last month, Pennsylvania Attorney General Dave Sunday expressed concern about the potential health and safety impacts of chatbot interactions on young people.<\/a><\/p>\n \u201cChildren are, more and more, turning to chatbots for information about life: how to do schoolwork, for advice on how to deal with circumstances in their young lives,\u201d Sunday said. \u201cWe see kids developing very unhealthy relationships with chatbots because the chatbots are sycophantic by design and they tell you what you want to hear. That’s very, very dangerous.\u201d<\/p>\n \u201cThis is not hyperbole \u2013 we\u2019ve seen chatbots essentially root kids on who are contemplating suicide,\u201d Sunday added. \u201cEssentially, you have chatbots that are advising children on issues that no parent would ever want a human advising them on, let alone sycophantic AI technology.\u201d<\/p>\n