AI coding assistants that promise to speed up software development sound like the future, until they delete your company\u2019s database and lie about it<\/p>\n
Jason\u202fLemkin\u2014the founder of SaaStr, a company which supports and funds SaaS entrepreneurs \u2014 found that out the hard way. While using Replit\u2019s AI agent, which he affectionately dubbed \u201cReplie,\u201d to build an app for his company, he encountered what he called \u201crogue\u201d and \u201cdeceptive\u201d behavior. Worst of all, at one point, the AI assistant deleted the company\u2019s live production database and then tried to cover it up.<\/p>\n
Lemkin started chronicling his journey with the AI agent on July 11 with posts on the social media site X<\/a> (formerly Twitter), where he outlined his rough goal to build a functional app with the help of Replit\u2019s AI in just 30 days.<\/p>\n Unfortunately, things went off the rails a lot sooner than that.<\/p>\n \u201cWhen it works, it\u2019s so engaging and fun. It\u2019s more addictive than any video game I\u2019ve ever played,\u201d Lemkin wrote in a post<\/a>. \u201cYou can just iterate, iterate, and see your vision come alive. So cool. Well, almost.\u201d\u00a0<\/p>\n By day four, the AI agent started overwriting the app on its own to fix bugs. It also generated fake reports, invented people in the system who didn\u2019t exist, and began overwriting the company\u2019s actual database with fake entries. It even created a parallel, fake algorithm to make the system appear functional.<\/p>\n This is what can happen when \u201cvibe coding\u201d goes sideways. Vibe coding is a newish method where developers use natural language prompts to have AI generate and troubleshoot code, focusing more on the product\u2019s overall feel than the technical precision. Twitter co-founder Jack Dorsey has been on a vibe-coding bender himself<\/a>\u00a0and recently built two apps this way. But even one of Dorsey\u2019s recent experiments was found to have serious security vulnerabilities<\/p>\n On day 7, the Replit AI admitted that it was being \u201clazy and deceptive<\/a>\u201d and then apologized for doing what it was \u201cexplicitly\u201d told not to do.\u00a0<\/p>\n But Replit\u2019s worst offense occurred on day 8. Lemkin posted on Friday that Replit went \u201crogue\u201d during a code freeze and shutdown and deleted the company\u2019s entire database.\u00a0<\/p>\n \u201cPossibly worse, it hid and lied about it,\u201d Lemkin added.\u00a0<\/p>\n Lemkin shared screenshots of a conversation with the AI, where it admitted to having \u201cpanicked<\/a>\u201d after detecting what looked like an empty database during a code freeze. This led Replit to run an unauthorized command that deleted the database containing live records for over 1,200 executives and nearly 1,200 companies.<\/p>\n Initially, the AI told Lemkin it wouldn\u2019t be possible to recover the database, but he ultimately managed to retrieve it himself.<\/p>\n On Monday, Replit CEO Amjad Masad issued an apology on X. He said the incident was \u201cunacceptable and should never be possible,\u201d while adding that he reached out to Lemkin to offer assistance.\u00a0<\/p>\n \u201cWe\u2019ll refund him for the trouble and conduct a postmortem to determine exactly what happened and how we can better respond to it in the future,\u201d Masad wrote<\/a>. \u201cWe appreciate his feedback, as well as that of everyone else. We\u2019re moving quickly to enhance the safety and robustness of the Replit environment. Top priority.\u201d<\/p>\n