Anthropic researcher Jacob Coxon has quit the AI company, saying those building the technology “earnestly believe that it could kill us all by the end of the decade”.

Mr Coxon said he had spent the past three years doing pre-training research at Anthropic and OpenAI.

“Neither company is acting responsibly,” he posted on X.

“They are racing straight to self-improving superintelligence and gambling with our lives.”

Superintelligence is considered to be the stage of AI when machines surpass human performance in every field.

Anthropic Alignment Science lead Evan Hubinger posted a reply on X saying Mr Coxon was correct.

“We really do earnestly believe AI could kill all humans! I personally think it is >10 per cent within the next decade,” he wrote.

“I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

Mr Coxon is one of a growing number of AI experts to call on the industry to slow down.

In July more than 1,300 employees of frontier AI companies signed a statement saying there was a “real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems”.

The letter called on the US government to support an international effort to develop tools that can “deliberately pace the frontier of automated AI development”.

A man in a suit and shirt, with short dark hair and glasses, sits talking in front of a blue background and TV screen.

CEO and co-founder of Anthropic, Dario Amodei. (Reuters: Denis Balibouse)

Responding to the initiative, called Pacing the Frontier, Anthropic said in an X post: “Our own research on recursive self-improvement, published last month, points to the need for tools to deliberately pace the frontier of AI development ⁠so society can prepare”.

OpenAI also responded to the letter, posting on X: “We believe that, at some point in the future, AI acceleration ‌for ⁠frontier model development may be so high that the world will need to pace the rate of AI advancement.”

AI could pose ‘existential’ risk, UN warns

The “alignment problem” for superintelligence is that an AI system may not share human values.

Autonomous drones and AI called out by UN human rights chief

UN-rights chief has addressed a council in Geneva saying AI posed a threat to humanity. 

On Monday, United Nations rights chief Volker Turk also warned that artificial intelligence could pose a threat to humanity and pledged to press AI firms to reduce risks that his office said included disruptions to services, communications and democratic systems.

He called for “cast-iron guarantees” to secure AI in a speech ahead of the start of his unprecedented second term.

“I share the concerns of industry insiders that advanced AI could pose an existential risk to humanity,” Mr Turk told the body in Geneva.

“I am calling here, today, for an all-out effort to put cast-iron guarantees in place around the safety and security of AI before it is too late.”

 ABC/Reuters