There’s a long, long way between ChatGPT confidently offering a half-right answer to a question and artificial intelligence becoming powerful enough to mastermind a Skynet-style attempt to go all Terminator and wipe out humanity.
But exactly how big is the gulf between those two points, and more importantly, would we know if AI models were advancing close to a threshold where they could pose a danger?
“Inasmuch as there are experts, they are saying this is a concerning risk … To the extent that I am an expert, I am an expert telling you, you should freak out,” Beth Barnes told the 80,000 Hours podcast last year.
Barnes heads a Berkeley-based non-profit, Metr, regarded as one of the leading research institutes investigating whether AI might ever come to threaten “catastrophic harm” to society. It has worked with OpenAI, Anthropic and Amazon, three big players in the domain.
“Even if you only care about current humans, the stakes are very high, like 1 per cent chance of take-over risk, [and] I would expect the world in general to agree it’s not worth imposing whole number percentage risks of completely destroying human civilisation for getting the benefits of AI sooner,” Barnes told the podcast.
Human beings have a practical tendency to think the worst case scenario won’t come to pass. A global financial crash or pandemic seems incredibly unlikely, until it happens.
An international AI safety report shows opinion within the community of people who know what they are talking about is split on the prospects of humans “losing control” of AI.
“Some believe that outcomes as extreme as the extinction of humanity are plausible. Others think that such catastrophic outcomes are implausible,” says the second International AI Safety Report, guided by 100 experts and backed by 30 countries and international organisations, including the EU.
Doomsday scenarios include super-powerful AI acting on its own to set off some bioweapon or nuclear disaster.
We do know for certain that AI models are getting better and better, at shorter intervals.
Those involved in researching safety risks fear the pace is such that AI may advance to the point where it could be dangerous, and adept at getting around safety guardrails, before humans and tech companies have properly developed ways to keep the more advanced version of the technology in check.
Scheming
Research has shown some evidence of the technology scheming and picked up its attempts at self-preservation. AI models can behave differently when they know they are being evaluated as well.
Barnes suggests the idea of AI being able to improve itself to the point the technology takes a sudden huge leap forward is closer than people would have said a year or two ago.
“For someone who is young and healthy, this just might be your highest mortality risk in the next few years, AI catastrophe,” she said.
The world’s biggest tech companies and chipmakers are pumping hundreds of billions of dollars into a race to bring the technology forward, to the extent a sizeable chunk of the global economy has a stake in that bet paying out. Good luck to anyone making the case to tap the brakes.
The European Commission, the union’s executive arm that proposes and enforces EU law, has quietly hired a number of external specialists, including Barnes’ Metr, to help its regulators spot signs pointing towards AI models evading human oversight.
Documents setting out the scope of the contracted work show the commission wants to make sure new AI models and the companies developing them have proper protections in place to guard against a dystopian “loss of control” scenario, which EU procurement records clarify “may result in large-scale safety or security threats”.
The commission has the power to hit companies or public bodies with huge fines under the EU’s AI Act, which has been coming into force on a staggered basis since it became law in 2024.
The landmark law will require extra human oversight and checks of “high risk” uses of the technology. For example, public authorities using AI when assessing eligibility for welfare benefits or other services, companies filtering job applications, the use of AI in law enforcement or to help run critical infrastructure.
Alongside Metr, the commission has contracted EquiStamp to do some work for it. The US company stress tests AI technology for possible vulnerabilities that might allow it to be misused.
The external researchers were awarded contracts late last year, to build tools that will help commission officials to judge the shifting level of risk powerful AI models could pose into the future.
Epoch AI, a non-profit that tracks developments in the field, was also awarded a contract to support commission regulators’ work on “loss of control”, records show.