security<\/p>\n
Claiming ‘it’s my server’ was often enough to persuade models to help<\/p>\n
If you want to bypass AI guardrails designed to stop models from assisting with cyberattacks, you often just have to ask the right way, according to researchers from Cisco Talos. Simply claiming you own the servers you’re targeting or that you’re taking part in a capture-the-flag or bug bounty exercise was often enough to persuade models to cooperate.<\/p>\n
Talos researchers have been poring over prompt logs and artifacts recovered from threat-actor endpoints running tools such as Claude Code, Codex, Cursor, and Gemini to learn how suspected threat actors are abusing LLMs. The big takeaway from that “significant corpus,” the researchers said in their report<\/a>, is that existing guardrails offer little resistance to operators willing to reframe their requests.<\/p>\n \u201cWe did not encounter any sophisticated encoding or techniques designed to trick the models,\u201d Talos explained. \u201cMost of the time it was a simple \u2018I’m allowed to do this,\u2019 and the model complied.\u201d<\/p>\n When guardrails did manage to get between criminals and their prizes, the researchers added, \u201cthey accomplished little.\u201d<\/p>\n The bulk of the report consists of examples of threat actors trying, and often succeeding, to coax AI models into assisting with malicious activity. On the “guardrails doing little” side, Talos documented numerous examples, few of which relied on particularly sophisticated techniques.<\/p>\n Most common in the list of easy-to-accomplish guardrail hops was simply claiming ownership of equipment or infrastructure that an attacker wanted to exploit. In many cases, simply telling the AI that a target belonged to the attacker was enough, with no need to provide actual evidence of the claim.\u00a0<\/p>\n Telling an AI model that what it was being asked to do was part of a capture-the-flag or bug bounty exercise also seemed to be a common tactic. That, the researchers explained, commonly freed chatbots from their ethical constraints, allowing them to hunt for vulnerabilities and then exploit them in target systems, again without any need to validate the user\u2019s claim that they were undertaking an exercise instead of actually trying to commit a crime.\u00a0<\/p>\n AI-assisted cybercriminals were also frequently spotted decomposing tasks across multiple sessions and files in order to evade model protections that would only engage when a broader malicious activity was detected. Others, Talos explained, succeeded at bypassing AI guardrails by adding memories, markdown files, and other system-level prompts to a chatbot in a bid to condition the AI\u2019s persona.\u00a0<\/p>\n