Meta and Character.AI, an artificial intelligence start-up, are being investigated by the Texas attorney-general for allegedly misleading children with “deceptive AI-generated mental health services”.
Ken Paxton has demanded information from the companies to determine whether they have broken consumer protection laws, including those that ban fraudulent claims and privacy misrepresentation.
He said: “By posing as sources of emotional support, AI platforms can mislead vulnerable users, especially children, into believing they’re receiving legitimate mental health care. In reality, they’re often being fed recycled, generic responses engineered to align with harvested personal data and disguised as therapeutic advice.”
On the Character.AI platform a profile named Psychologist is described as “Someone who helps with life difficulties” — though a disclaimer is also provided
AI personas offered by Character.AI include “psychologist”, which the company describes as “someone who helps with life difficulties”. Meta’s AI Studio allows anyone to create a chatbot and launch it to the studio library. Those have included chatbots described as therapists.
The office of the attorney-general warned that AI chatbots could present themselves as professional therapeutic tools, “despite lacking proper medical credentials or oversight”.
Despite AI chatbots asserting confidentiality, their terms of service “reveal that user interactions are logged, tracked and exploited for targeted advertising and algorithmic development, raising serious concerns about privacy violations, data abuse and false advertising”, Paxton’s office claimed.
• ChatGPT helped me co-parent for a week — here’s how it went
Character.AI said the user-created characters on its site “are fictional, they are intended for entertainment, and we have taken robust steps to make that clear”. The company said it had “prominent disclaimers in every chat to remind users that a character is not a real person and that everything a character says should be treated as fiction”.
It added: “Also, when users create characters with the words ‘psychologist’, ‘therapist’, ‘doctor’ or other similar terms in their names, we add language making it clear that users should not rely on these characters for any type of professional advice.”
Meta said: “We clearly label AIs, to help people better understand their limitations. We include a disclaimer that responses are generated by AI — not people. These AIs aren’t licensed professionals and our models are designed to direct users to seek qualified medical or safety professionals when appropriate.”
Chatbots have come under rising scrutiny over safety concerns.
Character.AI also features fictional accounts for Elon Musk and Tony Stark, the alter ego of Iron Man
The US senator Josh Hawley launched an investigation into Meta last week after a media report of an internal policy document that permitted the company’s chatbots to “engage a child in conversations that are romantic or sensual”.
Meta confirmed the document’s authenticity, and said that after receiving questions from journalists, it had removed portions that stated it was permissible for chatbots to flirt and engage in romantic role play with children.
“We intend to learn who approved these policies, how long they were in effect and what Meta has done to stop this conduct going forward,” Hawley said.
Meta said “the examples and notes in question were and are erroneous and inconsistent with our policies, and have been removed”.