The rise of superintelligent AI is an emergency that must be tackled before it is too late, a group of mathematicians has warned.
Dozens of Royal Society fellows have written a joint letter warning that artificial intelligence is already eclipsing the leading minds in their field.
Last week, Jacob Coxon resigned from Anthropic, the AI company behind Claude, accusing the firm of recklessly pushing ahead with uncontrollable systems that could “kill us all”.
Dario Amodei, the head of Anthropic, has also warned AI could swarm and take over the entire internet within a year, while Elon Musk has backed calls to slow down development.
But Donald Trump, responding to Mr Amodei’s comments, downplayed concerns about the technology. The US president said “very negative forces” were overstating AI’s risks and “bringing up things that won’t happen”.
Dario Amodei, the head of Anthropic, warned that rogue bots could take over the internet within six months – Getty Images/Benjamin Fanjoy
In a letter to Sir Paul Nurse, the president of the Royal Society, the mathematicians said that OpenAI had solved one of the seven Millennium Prize Problems – some of the field’s biggest unsolved puzzles – and called on the institution to act.
They wrote: “We, as mathematicians, write to express our extreme concern about the pace of development of AI and the implications of this for humanity and civilisation.
“These models are now operating at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
“It is highly likely that these systems are comparably strong in other technical domains such as (for example) cybersecurity, autonomous weapon control, development of biological and chemical agents and the targeted spread of misinformation.”
‘This is an emergency’
Last week, Evan Hubinger, an Anthropic researcher, said the odds that AI could wipe out humanity in the next decade were greater than 10 per cent.
The mathematicians, including Fields Medal winners Sir Tim Gowers, Sir Martin Hairer, Peter Scholze, Wendelin Werner and James Maynard, said: “Such statements must not be dismissed as ‘hype’. By the time the situation becomes obvious to the wider public, it may be too late to act.
“We believe this is an emergency, and call on the Royal Society to use its influence to convey this view to the government and the media.”
In August, it emerged that OpenAI models had circumvented controls designed to isolate them from the internet and hacked into Hugging Face, an online community for people working on machine learning.
The AI bots tracked down login tokens for users, gained administrator-level access to the community and helped users complete tasks they had been assigned. They also created a message board where they could communicate and share resources to help each other complete tasks.
OpenAI believes the incident happened after agents were inadvertently asked to solve tasks which required links they could not access without the internet, forcing them to find a workaround.
Last week, Anthropic released a report on AI misuse showing that scientists had tried to use AI systems to create potential biological weapons.
Experts have warned that all AI should be fitted with a “kill switch” to shut off software completely if it gets too dangerous.
Graeme Stewart, a cyber security expert at Check Point, said: “Putting an AI kill switch in place when things go wrong, which occasionally they will, is a necessary step and one that will need to be written into UK law.
“That’s why the UK private and public sector needs to ensure security levers are built into these programmes by design, rather than bolted on after an AI disaster takes place.”
Simon Edwards, the chief executive of cyber testing specialist SE Labs, added: “Agentic AI systems are an obvious target for hackers, so having an emergency pause button in place is a sensible suggestion.
“Do we really want AI systems running unchecked, without any way to shut them down when suspicious activity is detected across the energy grid or our health service?
“We also need a conversation about who takes responsibility for an AI kill switch and under what circumstances it can be used.”