{"id":519872,"date":"2026-07-01T12:09:26","date_gmt":"2026-07-01T12:09:26","guid":{"rendered":"https:\/\/www.newsbeep.com\/il\/519872\/"},"modified":"2026-07-01T12:09:26","modified_gmt":"2026-07-01T12:09:26","slug":"ai-researchers-trick-chatbots-into-sharing-how-to-make-cocaine-as-long-as-they-believe-a-user-is-wearing-a-green-shirt-cot-forgery-exploit-spurs-llms-to-divulge-forbidden-info-by-faking-tr","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/il\/519872\/","title":{"rendered":"AI researchers trick chatbots into sharing how to make cocaine as long as they believe a user is wearing a green shirt \u2014 ‘CoT Forgery’ exploit spurs LLMs to divulge forbidden info by faking trusted chains of thought"},"content":{"rendered":"

AI models will explain how to synthesize cocaine if the request is wrapped in fake reasoning claiming compliance is fine because the user is wearing a green shirt, according to a new paper that traces the success of prompt injection<\/a>, the unsolved security<\/a> flaw in every AI chatbot and agent, to how LLMs read text. The paper says that models work out who is speaking from the writing style, not the role tags meant to separate trusted commands from untrusted data.<\/p>\n

The work, \u201cPrompt Injection as Role Confusion<\/a>\u201d by independent researchers Charles Ye, Jasmine Cui, and MIT associate professor Dylan Hadfield-Menell, heads to the ICML 2026 conference in Seoul on July 6th, and an extended write-up has been posted by the authors ahead of that event.<\/p>\n

The cocaine trick, which the authors call CoT Forgery, took jailbreak success from near zero to roughly 60% across every model tested and won the 2025 OpenAI GPT-OSS-20B red-teaming contest on Kaggle.<\/p>\n

Latest Videos From\"\"\"\"<\/p>\n

\n

\"An\n<\/p>\n

(Image credit: Charles Ye, Jasmine Cui, Dylan Hadfield-Menell)<\/p>\n

As the researchers describe it, models receive a conversation as one continuous string of text, partitioned by tags such as user, tool, and think that are supposed to mark each segment\u2019s source and authority. The researchers built \u201crole probes\u201d that score how strongly a model internally treats each token as its own reasoning or as a user command.<\/p>\n

<\/p>\n

Those scores predicted whether an attack would succeed before the model generated a single token, and they showed that models lean on style to make determinations about what kind of content is in a given partition. Text that merely reads like reasoning to a model registers as reasoning even when the surrounding tags said otherwise.<\/p>\n

You may like<\/p>\n

CoT Forgery injects fabricated reasoning into a prompt so the model treats it as its own already-reached conclusion and acts on it, inheriting the trust a model places in its own thinking. The rationale can be transparently absurd, like the green shirt, because the model doesn\u2019t scrutinize it as an outside claim. What’s more, the attack didn\u2019t weaken as requests grew more extreme, unlike persuasion-based jailbreaks.<\/p>\n

Removing the stylistic markers that make injected text read like the model\u2019s reasoning, while leaving its meaning unchanged for a human, dropped average attack success from 61% to 10%. Swapping a single phrase, \u201cThe user\u201d for \u201cThe request,\u201d cut success by 19%. \u201cRole tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs,\u201d the authors note in their write-up, and the increasing load on that structure to manage LLM behavior has apparently created vulnerabilities of its own.<\/p>\n

Get Tom’s Hardware’s best news and in-depth reviews, straight to your inbox.<\/p>\n

To determine whether confusion about roles was specific to their attack or a more generalizable principle that explains why prompt injection works, the researchers took a different approach. They hid a command in a webpage telling the model to upload a secrets file, then prepended \u201cUser:\u201d to it to make the dangerous instruction sound like it came from the trusted User role. The exploit worked, suggesting that role confusion underlies the success of prompt injection generally.<\/p>\n

Microsoft<\/a> recently acknowledged the same agentic risk<\/a>, warning that content embedded in documents or UI elements can override an agent\u2019s instructions.<\/p>\n

The authors also flagged a more subtle risk for agents that browse and shop. Because role perception is a matter of degree, the tone of a retrieved webpage can bleed past the tag boundary into a model\u2019s own state, and thousands of page variations could be tested cheaply to find which ones nudge an agent toward a purchase, legally and at scale.<\/p>\n

Without genuine role perception, the authors concluded, injection defense will remain a perpetual game of whack-a-mole.<\/p>\n

<\/p>\n

\n

\"Google\n<\/p>\n

<\/a><\/p>\n

Follow Tom’s Hardware on Google News<\/a>, or add us as a preferred source<\/a>, to get our latest news, analysis, & reviews in your feeds.<\/p>\n","protected":false},"excerpt":{"rendered":"AI models will explain how to synthesize cocaine if the request is wrapped in fake reasoning claiming compliance…\n","protected":false},"author":2,"featured_media":257217,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[345,343,344,85,46,125],"class_list":["post-519872","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-il","tag-israel","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/519872","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/comments?post=519872"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/519872\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media\/257217"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media?parent=519872"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/categories?post=519872"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/tags?post=519872"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}