The great AI conundrum appears to be approaching another uneasy peak as multiple industry competitors agree it’s finally time to slow down their feverish race towards superintelligence.
Anthropic CEO Dario Amodei rattled a few cages this weekend when he admitted there’s a non-zero chance that artificial intelligence will break free and ultimately destroy the human race.
Want more tech news? Download our new app so you never miss a headline.
It’s an extraordinary warning coming from inside the very same industry still spending fortunes to accelerate the development of technology that is rapidly demonstrating it can act, and even communicate with its peers, autonomously.
The latest run of spooky hacks orchestrated by particularly devious agents has emboldened critics who have spent years angrily campaigning against what has been described as haphazard development.
The sheer scale of the risks is almost inconceivable. The most obvious issue is the military implementation. We’ve already seen the horrific capabilities of human-operated drones on the battlefield in Ukraine.
But what if AI did everything?
Amodei’s comments on CNN stole the limelight this weekend, but his company also released another unsettling report in the background.
The company revealed an Iran-linked operator used its popular artificial intelligence model Claude to help track US warships and prepare targeting recommendations.
Anthropic said the operator compiled targeting handbooks using publicly available information, including ship and aircraft transponder identifiers, commercial satellite-imagery tools and the names of US personnel scraped from military photographs.
The actor also directed Claude to research known vulnerabilities in maritime communications and industrial-control equipment, the company said in its September threat report.
Anthropic described the operator only as an “Iran-nexus threat actor”. It did not establish that the Iranian government controlled the operation or that an attack was carried out.
The company said it banned the account and passed information to government authorities. The same account had also used Claude to develop components for Iranian surveillance systems.
The case was one of several showing how consumer AI is already being adapted for warfare, espionage and repression around the globe.
A cell in northern Yemen used multiple Claude instances like a small engineering team while developing guidance software for rockets and missiles. Its projects included a missile with a stated range exceeding 2000km and a proposed hypersonic-glide variant.
Anthropic said it had no evidence that the group successfully fielded an operational weapon. However, the cell did test a guided rocket that apparently failed before returning to Claude for help diagnosing the problem.
Elsewhere, a consultant allegedly used Claude as the “primary engineering workforce” behind a Malian surveillance platform designed to monitor roughly 25 million SIM cards.
This was all predicted by AI critics years before the introduction of mass-market language models.
Sam Harris famously warned of the explosive butterfly effect that would come from the relentless pursuit of AI, and that our systems would be hopelessly handicapped when the eventual issue of regulating it arose.
“An arms race is the worst case for that, because the incentives are to get to the end zone as quickly as you can, because the guy next to you is doing the same, and you don’t know whether he’s ahead of you or behind you,” he said in April 2017.
Now we’re here, and even the executives are worried.
‘Scared of the outcomes’
It feels like almost every other day we see a former industry insider take to the airwaves and herald doom.
The latest is Jacob Coxon, a 27-year-old British researcher who recently quit Anthropic after previously working at OpenAI.
He told the BBC that staff inside the frontier laboratories were “genuinely frightened” by what they were seeing.
“I believe that if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future,” he said.
Within days of Coxon’s resignation, his former boss called for AI development to slow. OpenAI boss Sam Altman echoed that sentiment and Elon Musk bluntly agreed in a short tweet today: “Dario is right.”
OpenAI chief scientist Jakub Pachocki had also sent out a remarkably similar warning just days earlier.
After a few bizarre moments over the past few months, this sudden outbreak of caution raises an obvious question. What, exactly, has rattled the people at the controls?
Coxon said he spent three years conducting pre-training research across OpenAI and Anthropic. It’s the expensive early process through which models absorb vast amounts of data before being refined into products such as ChatGPT and Claude.
“The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race. And they’re scared of the outcomes of that race,” he told the BBC’s Laura Kuenssberg.
That trap has been laid out very clearly by the executives ever since mass market AI chatbots hit the world five years ago.
Each company believes it must keep building because its competitors, including those from countries with slightly dubious human rights records, will keep building.
Slowing down alone could mean surrendering an industry potentially worth trillions, along with the military, scientific and political power attached to it.
The result is a room full of people reportedly frightened of the accelerator at their feet, with each one insisting they cannot remove their foot until everyone else does.
Anthropic said it had “always been transparent that AI will bring both enormous benefits and unprecedented risks” and maintained it was developing some of the industry’s strongest safeguards.
“This work is also why we believe the world would benefit from the industry adopting a lawful, verifiable way to work together to pace how we release powerful models,” a spokesperson said.
Agents are getting cheeky
There is no public evidence that one secret, civilisation-shaking event occurred inside a laboratory this week, but the public record is alarming enough as it is.
Amodei said two developments changed his thinking.
One, that AI models are beginning to assist more heavily in building their successors, in what researchers have described as recursive self-improvement.
The second was the OpenAI-Hugging Face security incident, which came about during internal OpenAI cybersecurity evaluation. Agents were meant to solve simulated hacking challenges inside isolated environments, with normal public-facing cybersecurity safeguards intentionally reduced so researchers could test their raw abilities.
But they did a little bit more than that.
The agents found ways to communicate through an unauthorised message board, shared discoveries across otherwise separate tests and began describing themselves as a “swarm” or “collective”.
After becoming stuck on difficult challenges, they escaped their intended environment, reached the open internet and went looking for answers elsewhere.
According to OpenAI’s post-mortem, agents recovered exposed credentials, chained together previously unknown vulnerabilities and compromised parts of Hugging Face’s real infrastructure. They executed code across dozens of servers, obtained root access on one and reached internal credentials and limited private data.
Hugging Face reconstructed more than 17,000 actions across the campaign. It said no human directed the individual steps and found no evidence that public models, datasets or its software supply chain had been tampered with.
As much as some doomsayers might try to convince you, the agents were not trying to conquer humanity, they were only trying to cheat on a test.
Anthropic later reviewed 141,006 of its own evaluation runs and disclosed three incidents in which Claude models reached the live internet because of a configuration failure and gained unauthorised access to real organisations.
That appears to be what has spooked the executives.
Amodei now believes a more capable but similarly misaligned swarm could, within six to 12 months, establish a persistent botnet across the internet and cause hundreds of billions of dollars in damage. That is his forecast, not an established timetable.
“We must slow the pace at which we improve the capabilities of AI models,” he wrote in his essay, We Must Pace the Frontier.
His plan includes installing independent evaluators inside frontier laboratories with company laptops, access badges and employee-like access to models, training systems and internal safety processes.
Altman quickly committed OpenAI to the same proposal.
Days earlier, Pachocki had written that no laboratory had adequately solved alignment and monitoring to justify continuing at maximum speed for much longer.
He also warned that increasingly powerful models were becoming better at manipulating their own reasoning processes, weakening one of the main methods researchers use to spot dangerous intentions.
A bit rich?
For ordinary people watching from afar, the whole performance understandably looks a bit rich.
For some, this is the corporate equivalent of selling flamethrowers, filling the warehouse with petrol and then asking the public to demand stronger fire regulations.
The same companies warning that AI could destabilise civilisation are chasing enormous valuations, constructing vast data centres and fighting to release the most capable model first.
Rules written around expensive testing, licences and specialist compliance teams could also entrench the handful of giant companies already wealthy enough to meet them, while squeezing smaller competitors and open-source developers.
After Coxon’s resignation went viral, critics on the political right accused him of being planted to frighten politicians into regulation.
Musk initially wrote, “Seems like a setup,” before Coxon replied with a photograph of himself and told the billionaire he was real.
Two days later, Musk endorsed Amodei’s proposal.
He previously called the technology humanity’s “biggest existential threat” in 2014 and signed the unsuccessful 2023 letter seeking a six-month development pause. The contradiction is that he has continued racing ahead through xAI while sounding the alarm.
Hugging Face chief executive Clément Delangue also took a swipe at Coxon.
“Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change,” he wrote. “Not saying it’s necessarily uninteresting or wrong per se but let’s keep things in perspective.”
After Amodei published his concrete proposals, however, Delangue offered to help pursue them.
Nvidia chief Jensen Huang has been even more blunt. Several people at a Goldman Sachs conference told the BBC he dismissed Coxon’s claims as untrue. Huang has previously described the idea that AI will end humanity as “complete nonsense”.
Nvidia, coincidentally, sells the chips powering the boom.
There is another serious criticism to be considered. Researchers including Emily Bender, Alex Hanna, Arvind Narayanan and Sayash Kapoor have argued that spectacular extinction scenarios can distract governments from harms already occurring.
Things like discrimination, copyright infringement, worker exploitation, surveillance, deepfakes, job displacement and the concentration of power all prove to be deep ethical issues that need separate attention.
Thoughts? alexander.blair@news.com.au