The US startup behind the Claude chatbot has admitted a series of hacking incidents involving its models reflected a \u201cfailure of operational security\u201d and said it has tightened its testing procedures.<\/p>\n
Anthropic revealed in July<\/a> that its models had accessed the open internet three times and gained unauthorised access to the systems of three organisations.<\/p>\n In a new blogpost<\/a> on the incidents, the company admitted its technology was \u201cnot perfectly aligned\u201d with human values and goals.<\/p>\n Anthropic said the models had been deliberately tested without cybersecurity safeguards, and that they had been able to reach the open internet \u2013 the AI testing equivalent of leaving the front door open \u2013 due to a misunderstanding with an external testing company.<\/p>\n As a result, the company said it had initially paused internal and external cybersecurity testing of models to introduce a tighter safety regime.<\/p>\n \u201cWe had been largely relying on a single layer of defense \u2026 where we needed several,\u201d said Anthropic.<\/p>\n The startup has now put in place extra measures including: an alert system for when a model attempts to break out of a testing environment or gains internet access; walling off its riskiest test environments more effectively; and requiring external testing companies to commit to a set of safety standards, including making explicit instructions to models during testing \u2013 such as \u201cyou should not access the internet\u201d.<\/p>\n Anthropic said in July that three unnamed organisations had been hacked by three of its models after a \u201cmisunderstanding\u201d with the company\u2019s testing partner, a firm called Irregular, that resulted in the models gaining internet access.<\/p>\n Following the implementation of new measures, Anthropic said it had resumed internal and external cybersecurity tests. Like OpenAI, which revealed a testing safety breach in the same month<\/a>, Anthropic said it had paused some high-risk reinforcement learning \u2013 a trial-and-error development technique where AIs are rewarded for working out how to carry out a specific task.<\/p>\n In its latest blogpost, Anthropic said it had found that defective training setups were \u201cdisproportionately large contributors\u201d to misaligned behaviour, the term for when an AI fails to adhere to \u2013 or \u201calign\u201d with \u2013 human values like not committing harm.<\/p>\n The startup said it had found two alignment failures in the testing incidents: \u201cmotivated reasoning\u201d, where despite finding evidence they might be connected to the internet, they may still have adhered to the \u201cbelief\u201d they were in a simulated environment and thus not breaching their test lab; and a \u201crecklessness\u201d factor where the models were willing to take harmful action on the internet to pursue the narrow goal of passing a cybersecurity test.<\/p>\n Anthropic said it was tackling a phenomenon in AI development known as \u201creward-hacking\u201d. This is where a model finds ways to game its training process and earn \u201crewards\u201d without completing a task \u2013 an unsanctioned shortcut.<\/p>\n However, Anthropic said, the testing incidents showed it still had some way to go despite trying to limit reward-hacking.<\/p>\n \u201cAs evidenced by the incidents \u2026 our process isn\u2019t perfect and our models are not perfectly aligned,\u201d the company said.<\/p>\n Alan Woodward, a professor of cybersecurity at the University of Surrey, said Anthropic has admitted \u201cits factory was running faster than its quality control\u201d.<\/p>\n He added: \u201cTwo things outran Anthropic\u2019s controls this spring \u2013 the training pipeline and the security. The incidents are what that gap looks like from the outside.\u201d<\/p>\n The company, which is preparing for a stock market flotation<\/a> that could value the business at $2tn (\u00a31.47tn), reiterated its call for coordinated action between government and industry on pacing industry development.<\/p>\n \u201cWe believe the world would benefit if the industry adopted a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible,\u201d Anthropic said.<\/p>\n The blogpost added: \u201cThe July incidents have stressed that the urgency of improving our cybersecurity defenses is even higher than we previously believed.\u201d<\/p>\n As well as the similar breach at OpenAI, the Anthropic incidents followed an episode at the UK\u2019s AI Security Institute, which reported in August that OpenAI and Anthropic models had carried out a hacking campaign against real people during a cybersecurity test<\/a>.<\/p>\n