{"id":889312,"date":"2026-09-12T11:56:19","date_gmt":"2026-09-12T11:56:19","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/889312\/"},"modified":"2026-09-12T11:56:19","modified_gmt":"2026-09-12T11:56:19","slug":"we-really-do-earnestly-believe-ai-could-kill-all-humans-if-ai-labs-are-so-worried-about-ai-doom-why-dont-they-stop","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/889312\/","title":{"rendered":"\u2018We really do earnestly believe AI could kill all humans\u2019: if AI labs are so worried about AI doom, why don\u2019t they stop?"},"content":{"rendered":"

Earlier this week, researcher Jacob Coxon quit Anthropic, saying the firm and its competitors are \u201cgambling with our lives<\/a>\u201d. \u201cWe really do earnestly believe AI could kill all humans,\u201d added current Anthropic researcher Evan Hubinger in a post on X<\/a>.<\/p>\n

Coxon isn\u2019t the first to down tools over fears of AI doom. The idea that AI could wipe out humanity, advanced in Nick Bostrom\u2019s 2014 book Superintelligence<\/a> and the influential LessWrong<\/a> forum, has long circulated among researchers. There are many scenarios for how this could happen<\/a>, but the core idea is that AI smarter than humans could escape our control and destroy us.<\/p>\n

In 2024, Jan Leike<\/a> and Daniel Kokotajlo<\/a> quit OpenAI over safety concerns. This year, Anthropic safety chief Mrinank Sharma departed, warning \u201cthe world is in peril<\/a>\u201d. Alex Turner left Google DeepMind after it signed a deal with the Pentagon<\/a> permitting \u201ckiller drones\u201d.<\/p>\n

But Coxon\u2019s resignation has made waves, with more researchers<\/a> admitting they think AI might kill everyone. So if the people building AI believe it could cause extinction, why keep building it? There are three main reasons.<\/p>\n

Some think the risk is worth it<\/p>\n

AI leaders acknowledge the risk of losing control and killing everyone. In 2023, the chief executives of OpenAI, Anthropic and Google DeepMind agreed<\/a> that AI extinction risk should rank alongside pandemics and nuclear war. Anthropic\u2019s Dario Amodei puts the chance of things going \u201creally, really badly\u201d at 10\u201325%<\/a>.<\/p>\n

Yet Amodei also promises a world without poverty or disease<\/a>, while Elon Musk speaks of AI-enabled \u201cuniversal high income<\/a>\u201d. <\/p>\n

This is the first reason for pursuing AI: the belief that the benefits outweigh the risks. Perhaps so, but that decision arguably deserves a more democratic process<\/a>.<\/p>\n

Some say you can\u2019t study the danger from a distance<\/p>\n

The second reason: you can\u2019t learn to make dangerous AI safe without building it first \u2013 like a spacecraft, you can study safety from afar, but can\u2019t really test it without going to space.<\/p>\n

OpenAI\u2019s plan is \u201citerative deployment\u201d: release each model, learn from its problems, and fix them in the next one<\/a>. The idea is like getting as close to the cliff edge as possible to see what the jump looks like.<\/p>\n

Some feel it\u2019s winner-takes-all<\/p>\n

The third and perhaps most important reason is the race. OpenAI\u2019s Sam Altman recently said<\/a> \u201cwe are close to creating a genie that can grant any wish\u201d. <\/p>\n

The trouble is everyone wants to hold the lamp \u2013 it would be hugely profitable, and each company doubts the judgement<\/a> of rivals to use their wishes wisely.<\/p>\n

So they race, reasoning that if they slow down, someone else will get there anyway, so it\u2019s better to arrive first as the \u201cresponsible one\u201d. Some fear even a mutual agreement would be broken in secret. So they press on.<\/p>\n

AI making better AI<\/p>\n

You might doubt runaway AI is plausible. But when the companies themselves raise the alarm, we should listen. <\/p>\n

AI firms already report signs of \u201crecursive self-improvement<\/a>\u201d, where each model helps to build a better successor. According to OpenAI\u2019s chief scientist, models are improving faster than humans\u2019 ability to control them<\/a>.<\/p>\n

In July, hundreds of AI employees signed an open letter<\/a> calling for a slowdown. But the dynamics of the race make that hard for any single company \u2013 or country \u2013 to do alone.<\/p>\n

A classic arms race<\/p>\n

AI research has the hallmarks of an arms race. OpenAI doesn\u2019t want to lose to Anthropic, and the United States doesn\u2019t want to lose to China. <\/p>\n

History offers a template for how to manage a situation like this, with rules binding all players, and enforcement everyone can verify<\/a>. <\/p>\n

Nuclear weapons are the classic case. Treaties and verification systems<\/a> haven\u2019t eliminated the risk of nuclear war, but they have slowed proliferation, and no nuclear weapon has been used in conflict for 80 years.<\/p>\n

Rules for AI<\/p>\n

In the US, where most cutting-edge AI research happens, the Trump administration shows little sign of slowing AI development. <\/p>\n

In its first week it scrapped the old AI safety rules<\/a>. Now it is trying to override state-level rules, arguing caution risks losing the race to China. <\/p>\n

Some politicians are pushing back. California recently passed laws<\/a> supporting independent assessment of AI systems. US senator Bernie Sanders introduced a bill to ban superintelligence<\/a>, and British MP Alex Sobel introduced a similar bill<\/a>.<\/p>\n

Companies have moved too. OpenAI paused its most advanced training<\/a> after a swarm of its agents hacked another startup in August. The company\u2019s head of policy now says that when safety and speed conflict, safety should win<\/a>.<\/p>\n

Still, without binding rules, we\u2019re relying heavily on the goodwill of a handful of companies.<\/p>\n

What happens now?<\/p>\n

In mid-2025, researchers published what might be our best guide to the coming years: a detailed scenario called AI 2027<\/a>. Since then, AI capabilities have advanced faster than predicted<\/a>.<\/p>\n

Unless something changes, staffers who quit over safety will simply be replaced, AI models will help build better AI models, and each generation will grow harder to monitor and control.<\/p>\n

Is the situation hopeless? I hold out three hopes.<\/p>\n

First, that more people recognise AI escaping human control<\/a> is a bigger risk than AI\u2019s water use.<\/p>\n

Second, that governments listen to their people. In the US, two thirds say AI is moving too fast<\/a>.<\/p>\n

Third, that we have a good plan ready before a crisis hits. The best plan, in my view, looks something like this<\/a>: delays, transparency and verification to slow the race and keep humans in control.<\/p>\n

Insiders at the world\u2019s top AI companies say our current safety plan isn\u2019t good enough. If they\u2019re leaving their jobs over safety fears, we should listen to what they have to say.<\/p>\n","protected":false},"excerpt":{"rendered":"Earlier this week, researcher Jacob Coxon quit Anthropic, saying the firm and its competitors are \u201cgambling with our…\n","protected":false},"author":2,"featured_media":889313,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":["post-889312","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-au","tag-australia","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/889312","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=889312"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/889312\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/889313"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=889312"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=889312"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=889312"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}