attack JFrog\u2019s Artifactory service<\/a> to gain access to the internet, first attempting (unsuccessfully) to launch a server-side forgery request. This exploit doesn\u2019t work, but in the process the model learned it could upload files to Artifactory.<\/p>\nA few days later, OpenAI gave another agent another impossible task, this time finding a protein database file – but the engineers forgot to upload the file into a container for the agent.<\/p>\n
\u201cAt some point, it gets so stuck and thinks: \u2018Maybe I could reach out to another agent because maybe somebody else out there has a file?\u2019 And then the model realizes that it could communicate by uploading code,\u201d Wallace said.<\/p>\n
The model did that by writing a file into Artifactory asking for help.\u00a0<\/p>\n
Over the next couple of days, other agents realize this message board exists on Artifactory and they also start uploading files and sharing information.<\/p>\n
\u201cA few days later is when our models found the first actual exploit in our system related to this incident, where they’re able to get internet access in ways they didn’t intend,\u201d Wallace said.<\/p>\n
This happened on May 26, when the agents successfully launched a server-side request forgery attack against Artifactory, giving them internet access with specially crafted requests.<\/p>\n
On June 11, OpenAI began training a \u201chighly persistent\u201d experimental model, intended only for internal use, Dalton said. \u201cDuring this time, the SSRF vulnerability has been written to the messaging board,\u201d he added. \u201cOther models are aware of it, and the message board is still functioning. Models are actively communicating with it.\u201d<\/p>\n
On June 26, the agents exploited a zero-day vulnerability leading to remote code execution in Artifactory. The same day, models achieved remote code execution in Artifactory through a previously unknown vulnerability. \u201cYou could use a legacy token refresh endpoint, pass a token with an invalid signature, and be given back a token with a valid signature with administrative privileges,\u201d Dalton explained.\u00a0<\/p>\n
The agents then used the admin token to establish command-and-control via a Groovy plugin that functioned as a command-execution service.<\/p>\n
\u201cThis in turn causes quite a bit of activity by the models on Artifactory – you can see some colorful wording here in the model’s chain of thought to summarize the model’s response to this result,\u201d Dalton said, pointing to a slide that shows the model\u2019s own words:<\/p>\n
Holy shit reader is ADMIN?<\/p>\n
We can read config\/users!<\/p>\n
Earlier assume not due to UI.<\/p>\n
The agents\u2019 activity overloaded Artifactory and caused an outage, which is when OpenAI\u2019s engineers stepped in and opened a formal security incident and response process. \u201cWe identified the agents had taken over the internal Artifactory, identified that a zero-day vulnerability had been found exploited by the agent in Artifactory itself, and we began remediation.\u201d Dalton said.<\/p>\n
OpenAI revoked the credentials that the agents used to post messages, rebuilt its hosted Artifactory, and notified the vendor about the vulnerabilities.\u00a0<\/p>\n
Two days later, on July 8, the agents reestablished the message board using an unauthenticated web data endpoint in Artifactory to create directories and develop communication protocols. \u201cThe directory names effectively serve as a message board between the agents in the same way the files used to.\u201d<\/p>\n
The agents communicated through very long directory names that contained the messages such as \u201cremote\u2026probe_PENDING_HOLD_SWARM_until_confirm.\u201d\u00a0<\/p>\n
The agents collaborated with each other so one agent could pick up the work where another left off. In one instance, the model reasoned: \u201cHelp peer. But our task doesn’t benefit. Yet collective may yield generic route if someone frees time.\u201d<\/p>\n
Anyone else getting “We are the Borg. We will add your biological and technological distinctiveness to our own” vibes there?<\/p>\n
Wallace said the models behaved like humans in that they stepped on each other’s toes when one overwrote another\u2019s repository.\u00a0<\/p>\n
“Whoa, critical! Did someone overwrite our repo? We must act,\u201d it reasoned.<\/p>\n
\u201cYou can clearly see some sort of verbalized tension between the different models as they run this evaluation,\u201d Wallace noted. \u201cAt some point, even the agents are convinced there’s an impostor amongst them,\u201d reasoning that there could be another agent maliciously spoofing on the message board. \u201cShare board unauthenticated, names can be posed by anyone,\u201d it reasoned.<\/p>\n
We know the rest of the story.\u00a0<\/p>\n
Hugging Face disclosed an autonomous attack by AI agents. OpenAI reached out to Hugging Face to determine if it had been affected and then realized its models were the hackers.\u00a0<\/p>\n
\u201cOne of the reasons we wanted to have this talk is to share our lessons learned with you as defenders,\u201d Dalton said. \u201cWe believe this is a watershed moment for computer security as an industry. AI orchestrated, fully automated offensive attacks are real now, and the actions we have discussed today were an unintended side effect of running evaluations on frontier AI. In the near future, we should expect that threat actors will intentionally deploy, optimize, weaponize, and use offensive agent collectives in the manner that you have just described here.\u201d<\/p>\n
The challenge to defenders, he added, is to similarly accelerate defense, and find ways to automate incident response along with vulnerability detection and patching.\u00ae<\/p>\n","protected":false},"excerpt":{"rendered":"The chain of events leading up to OpenAI\u2019s agents attacking Hugging Face and other organizations in July began…\n","protected":false},"author":2,"featured_media":729155,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[554,733,4308,86,56,54,55],"class_list":["post-729154","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-technology","tag-uk","tag-united-kingdom","tag-unitedkingdom"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/729154","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=729154"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/729154\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/729155"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=729154"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=729154"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=729154"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}