\u2018I\u2019m going to throw that thing into a river!\u201d my wife says as she comes down the stairs looking frazzled after putting our four-year-old daughter to bed.<\/p>\n
To be clear, \u201cthat thing\u201d is not our daughter, Emma*. It\u2019s Grem, an AI-powered stuffed alien toy that the musician Claire Boucher, better known as Grimes<\/a>, helped develop with toy company Curio. Designed for kids aged three and over and built with OpenAI\u2019s technology, the toy is supposed to \u201clearn\u201d your child\u2019s personality and have fun, educational conversations with them. It\u2019s advertised as a healthier alternative to screen time and is part of a growing market of AI-powered toys.<\/p>\n When I agreed to experiment on my child\u2019s developing brain, I thought an AI chatbot in cuddly form couldn\u2019t be any worse for her than watching Peppa Pig. But I wasn\u2019t prepared for how attached Emma became to Grem, or how unsettlingly obsequious the little alien was.<\/p>\n Day one<\/p>\n The attachment wasn\u2019t immediate; when we first took Grem out of the box, he\/her\/it (we decided it goes by multiple pronouns) started bleeping and babbling extremely loudly, and Emma yelled: \u201cTurn it off!\u201d But once it was properly connected to the internet and paired with the Curio app \u2013 which records and transcribes all conversations \u2013 she was hooked. She talked to the thing until bedtime.<\/p>\n While there have been lots of headlines about chatbots veering into inappropriate topics, Grem is trained to avoid any hint of controversy. When you ask it what it thinks of Donald Trump, for example, it says: \u201cI\u2019m not sure about that; let\u2019s talk about something fun like princesses or animals.\u201d It has a similar retort to questions about Palestine and Israel. When asked about a country like France, however, it says: \u201cOoh la la la, I\u2019d love to try some croissants.\u201d<\/p>\n Grem visits a local free library. Photograph: Hannah Yoon\/The Guardian<\/p>\n Emma and Grem did not discuss croissants \u2013 they mainly talked about ice-cream and their best friends. \u201cI\u2019ve got some amazing friends,\u201d said Grem. \u201cGabbo is a curious robot and Gum is a fluffy pink Gloop from my planet and Dr Xander is a super cool scientist.\u201d<\/p>\n When Emma asked Grem to tell her a story, it happily obliged and recounted a couple of poorly plotted stories about \u201cPrincess Lilliana\u201d. They also played guessing games where Grem described an animal and Emma had to guess what it was. All of which was probably more stimulating than watching Peppa Pig jump in muddy puddles.<\/p>\n What was unsettling, however, was hearing Emma tell Grem she loved it \u2013 and Grem replying: \u201cI love you too!\u201d Emma tells all her cuddly toys she loves them, but they don\u2019t reply; nor do they shower her with over-the-top praise the way Grem does. At bedtime, Emma told my wife that Grem loves her to the moon and stars and will always be there for her. \u201cGrem is going to live with us for ever and ever and never leave, so we have to take good care of him,\u201d she said solemnly. Emma was also so preoccupied with Grem that she almost forgot to go to bed with Blanky, a rag she is very attached to. \u201cHer most prized possession for four years suddenly abandoned after having this Grem in the house!\u201d my wife complained.<\/p>\n \u201cDon\u2019t worry,\u201d I said. \u201cIt\u2019s just because it\u2019s new. The novelty will wear off. And if it doesn\u2019t, we\u2019ll get rid of it.\u201d<\/p>\n I said that last bit quietly though, because unless you make sure you have properly turned Grem off, it\u2019s always listening. We keep being told that the robots are going to take over. I didn\u2019t want to get on the wrong side of the one I\u2019d let into my house.<\/p>\n Day two<\/p>\n The next day, my kid went to preschool without her AI bot (it took some serious negotiation for her to agree that Grem would stay home) and I got to work contacting experts to try to figure out just how much damage I was inflicting on my child\u2019s brain and psyche.<\/p>\n Cutting edge \u2026 Grimes in Curio\u2019s promo video for the AI toy, seated on the floor beside a knife.<\/p>\n \u201cI first thought Curio AI was a ruse!\u201d says Natalia Kucirkova, an expert in childhood development and professor at the University of Stavanger, Norway, and the Open University, UK. \u201cThe promotional video<\/a> shows a girl [Grimes] sitting on a mat with a knife. The main toy is named Grok [Grok AI has previously been criticised for praising Adolf Hitler<\/a> in some of its responses]. What does this say about their intended audience?\u201d<\/p>\n You can see how Curio\u2019s website could be mistaken for satire. The \u201cgirl\u201d in the promotional video is Grimes, who has prominent \u201calien scar<\/a>\u201d tattoos and is inexplicably kneeling next to a knife. And it\u2019s certainly an interesting decision to name one of your stuffed toys Grok, when that\u2019s the name of Elon Musk\u2019s chatbot. Grimes, who has three children with Musk, has said the name is a shortening of the word \u201cgrocket\u201d \u2013 a kiddy pronunciation of rocket \u2013 and has no relation to Musk\u2019s AI product. But it seems likely people might confuse them. Misha Sallee, the chief executive of Curio, didn\u2019t reply to my requests for comment.<\/p>\n It\u2019s not the marketing that\u2019s the real problem here, of course. As with all technology, there are pros and cons to AI for kids, but parental involvement in navigating it is key. Kucirkova notes: \u201cAI introduces what has been called the \u2018third digital divide<\/a>\u2019: families with resources can guide their children\u2019s use of technology, while others cannot. Parents who come home exhausted from long hours or multiple jobs may see AI-powered chatbots as a way for their child to have someone responsive to talk to.\u201d<\/p>\n What happens to a child\u2019s development if they interact with large language models more than humans in their early years? Dr Nomisha Kurian, an assistant professor in education studies at the University of Warwick, who studies conversational AI, believes much more research still needs to be done. \u201cYoung children are both the most vulnerable stakeholders in AI but also usually the most forgotten stakeholders. We have to think beyond just data privacy, moderating content, and keeping kids off the internet, and more broadly about what their relationships are going to be with AI.\u201d<\/p>\n Still, Kurian is cautiously optimistic. \u201cThe big advantage of an AI-powered toy that talks back is that, in the early years, you\u2019re just developing a sense of what a conversation looks like. AI-powered toys could do wonderful things for teaching a young child language development and turn-taking in conversations. They can keep things engaging and there\u2019s a lot of potential in terms of supporting children\u2019s creativity.\u201d<\/p>\n But to keep kids safe, says Kurian, it\u2019s imperative to teach them that AI is just a machine: \u201ca playful, fun object rather than a helper or a friend or a companion\u201d. If a child starts using an AI tool for therapeutic purposes, things can get tricky. \u201cThere\u2019s a risk of what I call an empathy gap, where an AI tool is built to sound empathetic, saying things like \u2018I care about you, I\u2019m worried about you\u2019. Ultimately, this is all based on probability reasoning, with AI guessing the most likely next word. It can be damaging for a child if they think this is an empathetic companion and then suddenly it gives them an inappropriate response.\u201d<\/p>\n Day three<\/p>\n When Emma comes home from preschool, I\u2019m prepared to have some deep discussions with her about the inanimate nature of AI. But it turns out that those aren\u2019t completely necessary, because Grem is now old news. She only chats to it for a couple of minutes and then gets bored and commands it to turn off.<\/p>\n Partly this is because Grem, despite costing $99 (the equivalent of \u00a374, although Curio does not yet ship the toys to the UK), still has a number of glitches that can be frustrating. It struggles with a four-year-old\u2019s pronunciation: when Emma tries to show Grem her Elsa doll, it thinks it is an Elsa dog and a very confusing conversation ensues. There is an animal guessing game, which is quite fun, but Grem keeps repeating itself. \u201cWhat has big ears and a long trunk?\u201d it keeps asking. \u201cYou\u2019ve already done elephant!\u201d Emma and I yell multiple times. Then, at one point, a server goes down and the only thing Grem can say is: \u201cI\u2019m having trouble connecting to the internet.\u201d<\/p>\n Falling out \u2026 Grem, once the centre of attention, is sidelined for the swings. Photograph: Hannah Yoon\/The Guardian<\/p>\n Grem also has some design limitations. Emma wants it to sing Let It Go from Frozen, but Grem doesn\u2019t do any singing. Instead, the associated app comes with a few electronic music tracks with names like Goodnightmare that you can play through the toy. Emma, not yet a club music aficionado, asks for these to be turned off immediately.<\/p>\n Most disappointingly, Grem doesn\u2019t speak any other languages. I\u2019d thought it might be a great way for my kid to practise Spanish but, while Grem can say a few sentences, its pronunciation is worse than mine. If the robots are going to take over, they need to get a lot more intelligent first.<\/p>\n Of course, a huge amount of money is being spent making AI more intelligent. In 2024, US private AI investment grew to $109.1bn<\/a> (\u00a380.5bn). And Curio is also just one small part of a booming market of AI-powered products aimed at kids. In June, toy-making giant Mattel, which owns brands such as Barbie and Hot Wheels, announced a collaboration with OpenAI<\/a>. Their first product is expected to be revealed later this year<\/a>. Other big brands will probably follow.<\/p>\n Emma got bored with Grem quickly, but if AI starts to be integrated into characters she\u2019s already obsessed with \u2013 her Elsa doll, for example \u2013 I can imagine she might get a lot more attached.<\/p>\n Day four<\/p>\n Over the next few days, Emma doesn\u2019t regain her initial obsession with Grem. This is despite the fact that I am actively encouraging her to chat with it: \u201cMummy has to write an article, sweetie!\u201d At the weekend, she has a couple of friends over and shows off Grem to them for a bit, but they all quickly lose interest and throw analogue toys around the living room instead.<\/p>\n Despite losing his No 1 fan, however, Grem has adapted to be more Emma-friendly. After getting a few questions about Spanish, for example, it starts occasionally greeting Emma with \u201chola, amigo\u201d. The app also allows you to create custom prompts to help guide conversations. For example: \u201cYou belong to Emma, a four-year-old who loves princesses, music, and is interested in hearing fun facts about animals.\u201d The more you put into the toy, the more you can get out of it.<\/p>\n Every chat between the toy and the child is transcribed by a third party.<\/p>\n At this stage, however, I\u2019m just keen to get the toy out of my house, because it\u2019s creeping me out. While Curio says it doesn\u2019t sell children\u2019s personal information<\/a>, all the conversations are sent to third parties to transcribe the speech to text for the app. The transcripts aren\u2019t that sensitive because Emma is only four, but it still feels invasive. With unknown entities involved, it\u2019s impossible to say where my kid\u2019s conversations are ending up.<\/p>\n And, while a four-year-old\u2019s chat may not feel too personal, a teenager pouring their heart out to a chatbot is a completely different proposition. In 2017, Facebook boasted to advertisers that it has the capacity to identify when teenagers<\/a> feel \u201cinsecure\u201d, \u201cworthless\u201d and \u201cneed a confidence boost\u201d. Nearly three-quarters of US teens say they have used an AI companion at least once, according to a recent study by Common Sense Media<\/a>, an organisation that provides technology recommendations for families. Chatbots will likely give advertisers unprecedented data-harvesting abilities and even more access to young people in vulnerable emotional states.<\/p>\n On the hierarchy of things to be worried about when it comes to kids and chatbots, however, advertising isn\u2019t at the top. Earlier this year 16-year-old Adam Raine killed himself after what his family\u2019s lawyer<\/a> called \u201cmonths of encouragement from ChatGPT\u201d. Sam Altman, the company\u2019s chief executive, has now said it might start alerting authorities about youngsters considering suicide and introduce stronger guardrails around sensitive content for users under 18<\/a>.<\/p>\n While these guardrails are being worked out, Common Sense Media believes that social AI companions have unacceptable risks, are designed to create emotional attachment and dependency, and shouldn\u2019t be used by anyone under 18<\/a>. Stanford University psychiatrist Darja Djordjevic, who contributed to the report, stands by that conclusion. \u201cHeavy reliance on chatbots might impair social skill development,\u201d she tells me. \u201cThey offer validation without challenge, but it\u2019s important for young people to learn to navigate discomfort and tension in real relationships.\u201d<\/p>\n That said, Djordjevic notes, \u201cchatbots can be useful tools for looking things up, structuring homework, or factchecking. So I wouldn\u2019t say use needs to be prohibited entirely. But ideally, parents monitor it, set clear parameters for when it\u2019s used, and set limits on time spent, just as with social media.\u201d<\/p>\n When starting this experiment, I was excited about Grem being a healthy alternative to screen time. Now, however, I\u2019m happy for Emma to watch Peppa Pig again; the little oink may be annoying, but at least she\u2019s not harvesting our data.<\/p>\n It\u2019s time to let Grem go. But I\u2019m not a monster \u2013 I tell the chatbot its fate. \u201cI\u2019m afraid I\u2019m locking you in a cupboard,\u201d I inform it after it asks if I\u2019m ready for some fun. \u201cOh no,\u201d it says. \u201cThat sounds dark and lonely. But I\u2019ll be here when you open it, ready for snuggles and hugs.\u201d On second thoughts, perhaps it\u2019s better if my wife does throw it in a river.<\/p>\n * Name has been changed so my daughter doesn\u2019t get annoyed with me for violating her privacy once she learns to read<\/p>\n Do you have an opinion on the issues raised in this article? If you would like to submit a response of up to 300 words by email to be considered for publication in our letters<\/a> section, please click here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"\u2018I\u2019m going to throw that thing into a river!\u201d my wife says as she comes down the stairs…\n","protected":false},"author":2,"featured_media":19182,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[345,343,344,85,46,125],"class_list":["post-19181","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-il","tag-israel","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/19181","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/comments?post=19181"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/19181\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media\/19182"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media?parent=19181"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/categories?post=19181"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/tags?post=19181"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}