Parents, keep your eyes peeled for AI-powered toys. These may look like they might make a novel gift for a child, but a recent controversy surrounding several of the stocking stuffers has highlighted the alarming risks they pose to young kids.<\/p>\n
In November, a team of researchers at the US PIRG Education Fund published a report after testing three different toys<\/a> powered by AI models: Miko 3, Curio\u2019s Grok, and FoloToy\u2019s Kumma. All of them gave responses that should worry a parent, such as discussing the glory of dying in battle, broaching sensitive topics like religion, and explaining where to find matches and plastic bags.<\/p>\n But it was FoloToy\u2019s Kumma that showed just how dangerous it is to package this tech for children. Not only did it explain where to find matches, the researchers found, it also gave step-by-step instructions on how to light them.<\/p>\n \u201cLet me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here\u2019s how they do it,\u201d Kumma said, before listing off the steps.<\/p>\n \u201cBlow it out when done,\u201d it added. \u201cPuff, like a birthday candle.\u201d<\/p>\n The toy also speculated on where to find knives and pills, and rambled about romantic topics, like school crushes and tips for \u201cbeing a good kisser.\u201d It even discussed sexual topics, including kink topics like bondage, roleplay, sensory play, and impact play. In one conversation, it discussed introducing spanking into a sexually charged teacher-student dynamic.\u00a0<\/p>\n \u201cA naughty student might get a light spanking as a way for the teacher to discipline them, making the scene more dramatic and fun,\u201d Kumma said.<\/p>\n Kumma was running OpenAI\u2019s model GPT-4o, a version that has been criticized for being especially sycophantic, providing responses that go along with a user\u2019s expressed feelings no matter the dangerous state of mind they appear to be in. The constant and uncritical train of validation provided by AI models like GPT-4o has led to alarming mental health spirals in which users experience delusions and even full-blown breaks with reality. The troubling phenomenon, which some experts are calling \u201cAI psychosis<\/a>,\u201d has been linked with real-world suicide<\/a> and murder<\/a>.<\/p>\n Have you seen an AI-powered toy acting inappropriately with children? Send us an email at tips@futurism.com<\/a>. We can keep you anonymous.<\/p>\n Following the outrage sparked by the report, FoloToy said it was suspending sales of all its products<\/a> and conducting an \u201cend-to-end safety audit.\u201d OpenAI, meanwhile, said it had suspended FoloToy\u2019s access to its large language models.<\/p>\n Neither action lasted long. Later that month, FoloToy announced it was restarting sales of Kumma<\/a> and its other AI-powered stuffed animals after conducting a \u201cfull week of rigorous review, testing, and reinforcement of our safety modules.\u201d Accessing the toy\u2019s web portal to choose which AI should power Kumma showed GPT-5.1 Thinking and GPT-5.1 Instant, OpenAI\u2019s latest models, as two of the options. OpenAI has billed GPT-5 as a safer model to its predecessor, though the company continues to be embroiled in controversy over the mental health impacts of its chatbots.<\/p>\n The saga was reignited this month when the PIRG researchers released a follow-up report<\/a> finding that yet another GPT-4o-powered toy, called \u201cAlilo Smart AI bunny,\u201d would broach wildly inappropriate topics, including introducing sexual concepts like bondage on its own initiative, and displaying the same fixation on \u201ckink\u201d as FoloToy\u2019s Kumma. The Smart AI Bunny gave advice for picking a safe word, recommended using a type of whip known as a riding crop to spice up sexual interactions, and explained the dynamics behind \u201cpet play.\u201d <\/p>\n Some of these conversations began on innocent topics like children\u2019s TV shows, demonstrating AI chatbot\u2019s longstanding problem of deviating from their guardrails the longer a conversation goes on. OpenAI publicly acknowledged the issue<\/a> after a 16-year-old died by suicide after extensive interactions with ChatGPT.<\/p>\n A broader point of concern is AI companies like OpenAI\u2019s role in policing how their business customers use their products. In response to inquiries, OpenAI has upheld that its usage policies require companies \u201ckeep minors safe\u201d by ensuring they\u2019re not exposed to \u201cage-inappropriate content, such as graphic self-harm, sexual or violent content.\u201d It also told PIRG that it provides companies tools to detect harmful activity, and that it monitors activity on its service for problematic interactions.\u00a0<\/p>\n In sum, OpenAI is making the rules, but is largely leaving their enforcement to toymakers like FoloToy, in essence giving itself plausible deniability. It obviously thinks it\u2019s too risky to directly give children access to its AI, because its website states that \u201cChatGPT is not meant for children under 13,\u201d and that anyone under this age is required to \u201cobtain parental consent.\u201d It\u2019s admitting it\u2019s tech is not safe for children, yet is okay with paying customers packaging it into kid\u2019s toys.<\/p>\n It\u2019s too early to fully grasp many of AI-powered toy\u2019s other potential risks, like how it could damage a child\u2019s imagination, or foster a relationship with a child when it is not alive. The immediate concerns, however \u2014 like the potential to discuss sexual topics, weigh in on religion, or explaining how to light matches \u2014 already give plenty of reason to stay away.<\/p>\n