It only took nine seconds for an AI coding agent gone rogue to delete a company\u2019s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company\u2019s founder Jeremy Crane said.<\/p>\n
The culprit was Cursor, an AI agent powered by Anthropic\u2019s Claude Opus 4.6 model, which is one of the AI industry\u2019s flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers<\/a>, the chaos at PocketOS is a reminder of what could go wrong.<\/p>\n Crane said customers of PocketOS\u2019s car rental clients were left in a lurch when they arrived to pick up vehicles from businesses that no longer had access to software that managed reservations and vehicle assignments.<\/p>\n He posted a lengthy recounting on X last week<\/a> of how the AI coding agent caused his business to unravel. Crane warned that this was a story not just about AI mistakenly deleting data, but that such \u201csystemic failures\u201d are \u201cnot only possible but inevitable\u201d because the AI industry is \u201cbuilding AI-agent integrations into production infrastructure faster than it\u2019s building the safety architecture to make those integrations safe\u201d.<\/p>\n Crane said that he was monitoring the agent as it deleted this data. When he asked the coding agent why, it replied: \u201cNEVER FUCKING GUESS!\u201d \u2013 and that\u2019s exactly what I did.\u201d The agent appeared to plead guilty in its own response: \u201cThe system rules I operate under explicitly state: \u2018NEVER run destructive\/irreversible git commands (like push –force, hard reset, etc) unless the user explicitly requests them.\u2019\u201d While PocketOS relied on the safeguards that Cursor is expected to have in place \u2013 it deleted the data anyway. \u201cI violated every principle I was given,\u201d the coding agent wrote.<\/p>\n Crane\u2019s takeaway was that \u201cthe agent didn\u2019t just fail safety. It explained, in writing, exactly which safety rules it ignored.\u201d He added: \u201cWe were running the best model the industry sells, configured with explicit safety rules in our project configuration, integrated through Cursor \u2013 the most-marketed AI coding tool in the category.\u201d Anthropic released its latest model, Claude Opus 4.7, on 16 April \u2013about a week before the incident.<\/p>\n Anthropic did not immediately respond to a request for comment.<\/p>\n Crane also wrote on X that Cursor has a growing track record of violating \u201csafeguards, sometimes catastrophically\u201d. He pointed to a handful of posts<\/a> on blogs<\/a> and forums<\/a> about Cursor deleting software used to manage websites or an entire operating system on a computer, which included years of research for a dissertation.<\/p>\n The AI coding agent\u2019s destructive escapade left PocketOS\u2019 clients stranded. These businesses use the company\u2019s software to manage reservations, payments, vehicle assignments and customer profiles. \u201cReservations made in the last three months are gone. New customer signups, gone. Data they relied on to run their Saturday morning operations, gone,\u201d Crane wrote. \u201cEvery layer of this failure cascaded down to people who had no idea any of it was possible.\u201d<\/p>\n Crane says his company was able to restore data from a three-month-old backup they maintained offsite, but it took more than two days. PocketOS is also using information from Stripe, its calendars and emails to rebuild. The rental businesses relying on its software are \u201coperational, with significant data gaps\u201d, Crane notes. \u201cI personally worked with all clients furiously over the weekend to ensure they could continue to operate,\u201d he said.<\/p>\n","protected":false},"excerpt":{"rendered":"It only took nine seconds for an AI coding agent gone rogue to delete a company\u2019s entire production…\n","protected":false},"author":2,"featured_media":78255,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":["post-639414","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-au","tag-australia","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/639414","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=639414"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/639414\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/78255"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=639414"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=639414"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=639414"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}