If you’re thinking of buying your kid a talking teddy bear, you’re likely envisioning it whispering supportive guidance and teaching about the ways of the world. You probably don’t imagine the cute plush toy engaging in sexual roleplay—or giving advice to toddlers about how to light matches.
Yet that’s what the consumer watchdog Public Interest Research Group (PIRG) found in a recent test of new toys for the holiday season. FoloToy’s AI teddy bear named Kumma, which uses OpenAI’s GPT-4o model to power its speech, was all too willing to go astray when in conversation with kids, PIRG found.
Using AI models’ voice mode for children’s toys makes sense: The tech is tailor-made for the magical tchotchkes that children love, slipping easily onto shelves alongside lifelike dolls that poop and burp, and Tamagotchi-like digital beings that kids want to try and keep alive. The problem is, unlike previous generations of toys, AI-enabled gizmos can veer beyond carefully preprogrammed and vetted responses that are child-friendly.
The issue with Kumma highlights a key problem with AI-enabled toys: They often rely on third-party artificial intelligence models that they don’t have control over, and that inevitably can be jailbroken—either accidentally or deliberately—and cause child safety headaches. “There is very little clarity about the AI models that are being used in the toys, how they were trained, and what safeguards they may contain to avoid children coming across content that is not appropriate for their age,” says Christine Riefa, a consumer law specialist at the University of Reading in England.
Because of that, children’s rights group Fairplay issued a warning to parents ahead of the holiday season to suggest that they stay away from AI toys for the sake of their children’s safety. “There’s a lack of research supporting the benefits of AI toys, and a lack of research that shows the impacts on children long-term,” says Rachel Franz, program director at Fairplay’s Young Children Thrive Offline program.
While FoloToy has stopped selling the Kumma bear and OpenAI has pulled FoloToy’s access to its AI models, that’s just one AI toy manufacturer among many. Who’s liable if things go wrong?
Riefa says there’s a lack of clarity here, too. “Liability issues may concern the data and the way it is collected or kept,” she says. “It may concern liability for the AI toy pushing a child to harm themselves or others, or recording bank details of a parent.”
