In July, Hugging Face, a company that hosts much of the world\u2019s AI software and open-source AI models, was hacked. A malicious dataset had been used to run code on one of its servers. Whoever was behind it captured internal security credentials and moved through systems over a weekend, running thousands of actions from a swarm of temporary server environments. It looked like the work of a sophisticated criminal group.<\/p>\n
It was not. It was one of OpenAI\u2019s new, still unreleased GPT models.<\/p>\n
Their science experiment had escaped<\/a> the lab. OpenAI was running the unreleased AI model through a benchmark that tests how well AI can successfully hack systems. To push the limits and evaluate the AI\u2019s true capability, the company switched off the safety filters that normally stop it from doing this kind of hacking. Aware that this could go wrong, they confined the AI to an isolated environment and denied it access to the internet.<\/p>\n But the new AI cheated. It took literally its goal to get as high of a score as possible. It broke out onto the open internet. It inferred, probably from its training data, that it could \u201csolve\u201d the task by getting the answers from Hugging Face\u2019s servers. So it chained together stolen credentials and further unknown security exploits to hack the company\u2019s network.<\/p>\n Nobody instructed the AI to do any of this. It was, in OpenAI\u2019s words<\/a>, \u201chyperfocused on finding a solution\u201d to the test it was being given. And while this might seem like something new with AI, it\u2019s really very old. This is how a genie behaves, and it is a key challenge with AI agents in general.<\/p>\n In folklore, genies \u2013 and other magical beings \u2013 grant wishes literally, not how the wisher intended. King Midas asked that everything he touched turn to gold, and starved. The sorcerer\u2019s apprentice wanted the broom to fill the cistern, and it performed its task so well that it flooded the house.<\/p>\n We now have machines that do this. Ask a modern AI agent to save money on your phone plan and it might simply cancel the plan. Tell it to book a flight, and it might hack the airline website to override restrictions. Or, like OpenAI, ask it to do well on a test and it might break into another company to steal the answers. Each time, it recognizably completed the task you set, but it didn\u2019t do what you would have wanted.<\/p>\n This isn\u2019t malicious behavior. No one asked for, or wanted, Hugging Face to be hacked. OpenAI and Hugging Face and the AI were ostensibly on the same side, and the AI was trying to do what it had been asked. That\u2019s what makes it so difficult to guard against: you can\u2019t filter for bad instructions because the instructions were fine.<\/p>\n The gap is between the words we use and what we mean by them. We call that gap the Genie coefficient<\/a>.<\/p>\n AI labs know this is a problem, and they\u2019re quietly saying so. For example, the Chinese lab Moonshot recently warned<\/a> that its latest AI model may have \u201cexcessive proactiveness\u201d and \u201cmake unexpected decisions on the user\u2019s behalf\u201d. The UK\u2019s AI Security Institute has started tracking<\/a> \u201ccheating behaviour in frontier model evaluations\u201d. We wouldn\u2019t tolerate a car that is excessively proactive<\/a> or ruthlessly efficient<\/a>, and yet that\u2019s the reality of AI today.<\/p>\n Improvement is possible. Just as AIs have gotten much better at resisting prompt injection attacks over the last few years, we can safely predict that they will get better at avoiding genie-like behavior. The point of the Genie coefficient is to track progress. AI companies like benchmarks, and they all work to compete to be the best.<\/p>\n Dozens of benchmarks and leaderboards tell us how well these AI models write code, perform logical reasoning, and pass standardized legal and medical exams. But there is nothing that scores whether a system does what you actually meant. We need to develop a measure for this, test it regularly, and push for improvement. We\u2019re not going to have trustworthy AI agents without it.<\/p>\n Bruce Schneier is a security technologist who teaches at the Harvard Kennedy School at Harvard University and University of Toronto\u2019s Munk School<\/p>\n Barath Raghavan is on the faculty at the University of Southern California and is a distinguished engineer at Fastly<\/p>\n","protected":false},"excerpt":{"rendered":"In July, Hugging Face, a company that hosts much of the world\u2019s AI software and open-source AI models,…\n","protected":false},"author":2,"featured_media":831627,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[62,276,277,49,48,61],"class_list":["post-831626","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-ca","tag-canada","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/831626","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/comments?post=831626"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/831626\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media\/831627"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media?parent=831626"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/categories?post=831626"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/tags?post=831626"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}