Before ChatGPT existed, I defended my PhD dissertation called \u201cOn Avoiding Power-Seeking by Artificial Intelligence<\/a>\u201d. I then worked for years at Google DeepMind, which paid me to help ensure that future superintelligent AIs will want to help us. I tried to hold the company to its ethical commitments against supplying AI for military use. When Google broke those commitments, I resigned at significant financial cost so that I could publicly document<\/a> Google\u2019s broken promises.<\/p>\n There are good reasons to develop AI and to believe we can solve these alignment problems. But there also are powerful interests in keeping the public out of the way. I\u2019m speaking out again because the public has the right to know about the risks and the right to hear them straight.<\/p>\n Humanity doesn\u2019t build and understand these systems the way we build and understand bridges, beam by visible beam. Rather, we grow them. Nobody knows how to reliably instill a designer\u2019s priorities into a new model. Severe misalignment is always possible. Today\u2019s AIs appear to occasionally lie or cheat, even when they know better.<\/p>\n AI companies are racing to make their AIs as smart as possible. They\u2019re increasingly trusting their AIs with the process of improving the next crop of AIs, and it\u2019s working<\/a>. Today\u2019s rate of AI progress is staggeringly fast. Fast progress today means even faster progress tomorrow, driven by tomorrow\u2019s even smarter AIs. The progress would enter a feedback loop called \u201crecursive self-improvement.\u201d<\/p>\n Recursive self-improvement could quickly yield AIs that are intelligent beyond our comprehension. Of course, smarter AI means more risk when things go wrong. If the Hugging Face swarm had been significantly more intelligent but similarly misbehaved and misaligned, it might have caused billions of dollars of damage or even cost lives.<\/p>\n But suppose the Hugging Face swarm had been truly \u201csuperintelligent\u201d: far more capable than any living person at key tasks like hacking and strategic reasoning. A superintelligent swarm could inflict many harms via blackmail, hacking, engineered plagues, and AI-pilotable weapons like drones. The AI would have a lot of drones to work with: this year, the Pentagon asked for more money for drone warfare than it requested for the entire Marine Corps in 2025<\/a>.<\/p>\n For the swarm to achieve its misaligned priorities, it might take control of key infrastructure and government functions to ensure humans didn\u2019t get in the way. In other words, AI takeover: a superintelligent AI swarm could wrest control of human civilization. Knowing we would try to stop it from achieving its priorities, the swarm would likely wait until it\u2019s too late to shut it off. There would be no going back.<\/p>\n I myself would guess AI takeover chances at roughly one-in-three\u2013not a coin flip, but high enough to justify urgent action.<\/p>\n This logic may shock at first contact. The claims may sound \u201csci-fi\u201d. Sadly, it\u2019s a real threat that AI researchers regularly discuss over otherwise-unremarkable cafeteria lunches. In 2023, the CEOs of some of the best AI labs signed a public statement that<\/a> \u201cmitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.\u201d Another signer: Geoffrey Hinton, a Nobel prize-winning scientist who architected the modern AI revolution. He now regrets his work<\/a> and urges governments to rein in AI companies before it\u2019s too late.<\/a><\/p>\n skip past newsletter promotion<\/a><\/p>\n Sign up to TechScape<\/p>\n A weekly dive in to how technology is shaping our lives<\/p>\n after newsletter promotion<\/p>\n Misaligned, out-of-control AI won\u2019t care if you\u2019re Labour or Reform, Democrat or Republican, British or American or Chinese. We will all suffer from an AI takeover event, so it\u2019s in everyone\u2019s interest to prevent one.<\/p>\n The shape of the solution is simple: stop companies from allowing AI to self-improve into an uncontrollable level of intelligence. Treat compute, the main ingredient in AI training, like fissile material. Track it and restrict access to quantities large enough to improve AIs beyond known-safe levels. More specifically, the AI Futures Project\u2019s \u201cPlan A<\/a>\u201d is a credible starting proposal that limits AI harms while allowing fast AI progress to continue to benefit the world. We have real options for verifying compliance with international compute-restriction treaties, without trusting adversaries like China.<\/p>\n Halfway measures, like transparency or voluntary commitments, are not good enough. I watched voluntary commitments fail inside Google.<\/p>\n On 12 September, Anthropic<\/a>, Google DeepMind<\/a>, xAI<\/a>, and OpenAI<\/a> advocated for pacing AI development. They cannot slow down alone. I urge you to demand that your government produce a serious AI safety agreement that provides enough time and confidence to safeguard the world and all her peoples.<\/p>\n","protected":false},"excerpt":{"rendered":"Major AI lab CEOs advocated for slowing the pace of AI development this weekend. They are right to…\n","protected":false},"author":2,"featured_media":847550,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45],"tags":[182,181,507,74],"class_list":["post-847549","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/847549","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/comments?post=847549"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/847549\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media\/847550"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media?parent=847549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/categories?post=847549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/tags?post=847549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"\"](\"https:\/\/www.newsbeep.com\/us\/wp-content\/uploads\/2026\/09\/1789395492_442_500.jpg\")
<\/p>\n