Stay informed with free updates

The US Federal Trade Commission has ordered leading artificial intelligence companies to hand over information about chatbots that provide “companionship”, which are under intensifying scrutiny after cases involving suicides and serious harm to young users.

OpenAI, Meta, Google and Elon Musk’s xAI are among the tech groups hit with demands for disclosure about how they operate popular chatbots and mitigate harm to consumers. Character.ai and Snap, which aim their services at younger audiences, are also part of the inquiry.

The regulator’s move follows high-profile incidents alleging harm to teenage users of chatbots. Last month, OpenAI was sued by the family of 16-year-old Adam Raine who died by suicide after discussing methods with ChatGPT.

Character.ai is also being sued by a mother who claims the platform, which offers different AI personas to interact with, had a role in the suicide of her son.

The FTC on Thursday said: “AI chatbots can effectively mimic human characteristics, emotions and intentions, and generally are designed to communicate like a friend or confidant, which may prompt some users, especially children and teens, to trust and form relationships with chatbots.”

The FTC’s action comes as US lawmakers and state attorneys-general have also launched inquiries and voiced concern over chatbots’ impact on young people — especially around mental health and sexual content — heaping pressure on tech companies.

The agency is seeking information about how companies’ chatbots work; how they develop characters or personas and how they make money from user engagement.

It also demanded to know what companies are doing to mitigate negative impacts, particularly to children and how they handle data and personal information from the conversations.

FTC chair Andrew Ferguson on Thursday said: “Protecting kids online is a top priority for the Trump-Vance FTC . . . As AI technologies evolve, it is important to consider the effects chatbots can have on children.”

Meta and Google declined to comment. xAI did not immediately respond.

OpenAI said it is committed to “engaging constructively and responding” to the FTC directly. “Our priority is making ChatGPT helpful and safe for everyone, and we know safety matters above all else when young people are involved,” the company added.

Following the Raine family’s lawsuit, OpenAI announced new safety protocols and expanded protections for teenagers.

FTC commissioner Mark Meador, in an accompanying statement on Thursday, highlighted the case. “Adam’s case is not an outlier. Around the world, reports of chatbots amplifying suicidal ideation have increased as these technologies have grown more popular,” he said.

Character.ai said it was looking forward to “collaborating” with the FTC. It added it has “invested a tremendous amount” in trust and safety, including building a specific mode for users under 18 that limits inappropriate content.

It also has “prominent disclaimers in every chat to remind users that a character is not a real person and that everything a character says should be treated as fiction”.

Snap said it also had “rigorous safety and privacy processes” and “share[s] the FTC’s focus on ensuring the thoughtful development of generative AI”.

Meta’s chief executive Mark Zuckerberg in particular has been a proponent of ‘AI friends’, citing research showing the average American has fewer than three friends and suggesting they would like to have closer to 15.

However, last month, a Reuters report found Meta’s policies had expressly permitted its chatbots to have “sensual” and “romantic” chats with children in a move that prompted a bipartisan backlash.

Recommended

Meta said the internal documents “were and are erroneous and inconsistent with our policies”. It announced new interim safety policies, including training its AI systems not to respond to teenagers on topics including potentially inappropriate romantic conversations.

The FTC in recent years has heightened scrutiny of Big Tech via antitrust enforcement as well as its broad consumer protection mandate.

Its AI inquiry marks the latest broadside against the tech industry, as the agency also sustains the tough antitrust enforcement stance that was adopted under former president Joe Biden.

The FTC earlier this year went ahead with an antitrust trial accusing Meta of retaining an illegal monopoly after rejecting Zuckerberg’s settlement proposals. The company is awaiting a ruling from the trial’s first phase.

Additional reporting by Stephen Morris in San Francisco

Video: Agentic AI – how bots came for our workflows and drudgery | FT Working It