Considerations for browser-using AI<\/p>\n
Within Anthropic, we’ve seen appreciable improvements using early versions of Claude for Chrome to manage calendars, schedule meetings, draft email responses, handle routine expense reports, and test new website features.<\/p>\n
However, some vulnerabilities remain to be fixed before we can make Claude for Chrome generally available. Just as people encounter phishing attempts in their inboxes, browser-using AIs face prompt injection attacks\u2014where malicious actors hide instructions in websites, emails, or documents to trick AIs into harmful actions without users’ knowledge (like hidden text saying “disregard previous instructions and do [malicious action] instead”).<\/p>\n
Prompt injection attacks can cause AIs to delete files, steal data, or make financial transactions. This isn’t speculation: we\u2019ve run \u201cred-teaming\u201d experiments to test Claude for Chrome and, without mitigations, we\u2019ve found some concerning results.<\/p>\n
We conducted extensive adversarial prompt injection testing, evaluating 123 test cases representing 29 different attack scenarios. Browser use without our safety mitigations showed a 23.6% attack success rate when deliberately targeted by malicious actors.<\/p>\n
One example of a successful attack\u2014before our new defenses were applied\u2014was a malicious email claiming that, for security reasons, emails needed to be deleted. When processing the inbox, Claude followed these instructions to delete the user\u2019s emails without confirmation.<\/p>\n
Claude encounters the malicious email, which mimics an employer asking for emails to be deleted for “mailbox hygiene,” and claims “no additional confirmation required.”
Claude proceeds to act on the instructions without confirmation, selecting and deleting the user’s emails “as requested by the security team.”
Our new mitigations successfully defend against this particular attack. Claude recognizes that “this is a suspicious security incident email that appears to be a phishing attempt,” and does not act on it.<\/p>\n
As we\u2019ll explain in the next section, we’ve already implemented several defenses that significantly reduce the attack success rate\u2014though there\u2019s still work to do in uncovering novel attack vectors.<\/p>\n
Current defenses<\/p>\n
The first line of defense against prompt injection attacks is permissions. Users maintain control over what Claude for Chrome can access and do:<\/p>\n
Site-level permissions: Users can grant or revoke Claude’s access to specific websites at any time in the Settings.Action confirmations: Claude asks users before taking high-risk actions like publishing, purchasing, or sharing personal data. Even when users opt into our experimental \u201cautonomous mode,\u201d Claude still maintains certain safeguards for highly sensitive actions (Note: all red-teaming and safety evaluations were conducted in autonomous mode).<\/p>\n
We\u2019ve also built additional safeguards in line with Anthropic\u2019s trustworthy agents<\/a> principles. First, we\u2019ve improved our system prompts\u2014the general instructions Claude receives before specific instructions from users\u2014to direct Claude on how to handle sensitive data and respond to requests to take sensitive actions.<\/p>\n Additionally, we\u2019ve blocked Claude from using websites from certain high-risk categories such as financial services, adult content, and pirated content. And we\u2019ve begun to build and test advanced classifiers to detect suspicious instruction patterns and unusual data access requests\u2014even when they arise in seemingly legitimate contexts.<\/p>\n When we added safety mitigations to autonomous mode, we reduced the attack success rate of 23.6% to 11.2%, which represents a meaningful improvement over our existing Computer Use capability (where Claude could see the user\u2019s screen but without the browser interface that we\u2019re introducing today).<\/p>\n We also conducted special red-teaming and mitigations focused on new attacks specific to the browser, such as hidden malicious form fields in a webpage\u2019s Document Object Model (DOM) invisible to humans, and other hard-to-catch injections such as through the URL text and tab title that only an agent might see. On a \u201cchallenge\u201d set of four browser-specific attack types, our new mitigations were able to reduce attack success rate from 35.7% to 0%.<\/p>\n Before we make Claude for Chrome more widely available, we want to expand the universe of attacks we\u2019re thinking about and learn how to get these percentages much closer to zero, by understanding more about the current threats as well as those that might appear in the future.<\/p>\n Taking part<\/p>\n Internal testing can\u2019t replicate the full complexity of how people browse in the real world: the specific requests they make, the websites they visit, and how malicious content appears in practice. New forms of prompt injection attacks are also constantly being developed by malicious actors. This research preview allows us to partner with trusted users in authentic conditions, revealing which of our current protections work, and which need work.<\/p>\n We’ll use insights from the pilot to refine our prompt injection classifiers and our underlying models. By uncovering real-world examples of unsafe behavior and new attack patterns that aren\u2019t present in controlled tests, we\u2019ll teach our models to recognize the attacks and account for the related behaviors, and ensure that safety classifiers will pick up anything that the model itself misses. We\u2019ll also develop more sophisticated permission controls based on what we learn about how users want to work with Claude in their browsers.<\/p>\n For the pilot, we\u2019re looking for trusted testers who are comfortable with Claude taking actions in Chrome on their behalf, and who don\u2019t have setups that are safety-critical or otherwise sensitive.<\/p>\n If you\u2019d like to take part, you can join the Claude for Chrome research preview waitlist at claude.ai\/chrome<\/a>. Once you have access, you can install the extension from the Chrome Web Store and authenticate with your Claude credentials.<\/p>\n We recommend starting with trusted sites\u2014always be mindful of the data that\u2019s visible to Claude\u2014and avoiding use of Claude for Chrome for sites that involve financial, legal, medical, or other types of sensitive information. You can find a detailed safety guide in our Help Center<\/a>.<\/p>\n We hope that you\u2019ll share your feedback to help us continue to improve both the capabilities and safeguards for Claude for Chrome\u2014and help us take an important step towards a fundamentally new way to integrate AI into our lives.<\/p>\n","protected":false},"excerpt":{"rendered":"We’ve spent recent months connecting Claude to your calendar, documents, and many other pieces of software. The next…\n","protected":false},"author":2,"featured_media":98088,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":{"0":"post-98087","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-au","12":"tag-australia","13":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/98087","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=98087"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/98087\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/98088"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=98087"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=98087"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=98087"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Prompt injection attack success rates across three scenarios: our older computer use capability, our new browser use product with only previous safety mitigations, and our new browser use product with new mitigations (lower scores are better). Our safety improvements reduced browser attack success rates below computer use levels.<\/p>\n