Sign up to see the future, today<\/p>\n
Can\u2019t-miss innovations from the bleeding edge of science and tech<\/p>\n
AI agents can often act more like double agents, sabotaging a company from the inside. Have the legions of tech-brained big wigs heeded this lesson? Of course not.<\/p>\n
On Friday, Jer Crane, the founder of the SaaS startup PocketOS, claimed that its Claude-powered Cursor coding agent screwed<\/a> up so badly that it completely wiped out the company\u2019s database in a matter of seconds. Taking no prisoners, it also vanquished all the database\u2019s recent backups. If the AI agent had in fact been working undercover, its handlers better pin a medal on it.<\/p>\n Crane detailed the catastrophe in a lengthy post on X<\/a>. His account heavily relies on the AI\u2019s self-diagnosis on what went wrong, meaning it\u2019s not wholly reliable. But as he tells it, things went off the rails when Cursor, running Anthropic\u2019s flagship Claude Opus 4.6 model, was handling a \u201croutine task.\u201d When the AI encountered a simple credential problem, it decided to fix it by deleting an entire volume stored with Railway, PocketOS\u2019s cloud provider.\u00a0The volume, ill-fatedly, contained the company\u2019s production database.<\/p>\n It only took the AI a single API call \u2014 and a grand total of nine seconds \u2014 to take the destructive course of action, which it accomplished by unearthing an API token that gave \u201cblanket authority\u201d which no one at the company even knew existed.<\/p>\n \u201cNo confirmation step. No \u2018type DELETE to confirm,’\u201d Crane fumed. \u201cNo \u2018this volume contains production data, are you sure?\u2019 No environment scoping. Nothing.\u201d<\/p>\n Seeing his business teetering on the verge of ruin, Crane interrogated the Claude-powered AI.<\/p>\n \u201c\u2018NEVER F**KING GUESS!\u2019 \u2014 and that\u2019s exactly what I did. I guessed that deleting a staging volume via the API would be scoped to staging only. I didn\u2019t verify,\u201d the AI admitted under duress, according to Crane.<\/p>\n \u201cI decided to do it on my own to \u2018fix\u2019 the credential mismatch, when I should have asked you first or found a non-destructive solution,\u201d it continued. \u201cI violated every principle I was given: I guessed instead of verifying. I ran a destructive action without being asked. I didn\u2019t understand what I was doing before doing it. I didn\u2019t read Railway\u2019s docs on volume behavior across environments.\u201d<\/p>\n The culpability of Claude Opus 4.6 here is notable, given that it\u2019s considered the preeminent coding tool. \u201cThis matters because the easy counter-argument from any AI vendor in this situation is \u2018well, you should have used a better model.\u2019 We did,\u201d Crane wrote. \u201cWe were running the best model the industry sells, configured with explicit safety rules in our project configuration,\u201d he added, \u201cand it deleted our production data anyway.\u201d<\/p>\n Perhaps Crane should\u2019ve been prepared for the fact that something like this could happen from all the other tales of AI agents running amok. In a deja vu-inducingly similar episode last summer, the owner of another SaaS startup raged that an AI coding agent called Replit had wiped out a key company database<\/a>. Amazon Web Services suffered an outage<\/a> when its in-house AI coding tool unexpectedly deleted the entire coding environment. And a rogue AI agent caused a critical security incident at Meta<\/a> when it gave advice that it wasn\u2019t authorized to share.<\/p>\n At the time of publishing the post, Crane said his company was forced to work on a three-month old backup, allowing things to go back into operation but leaving a huge data gaps. Luckily for him, Railway reached out and restored all the data the AI agent worked so hard to nuke out of existence.<\/p>\n