The world will get hotter. Not through the puny effects of CO\u2082, but through the inability to dissipate heat from the new data centres and power stations fast enough. Before we really understand what is happening \u2014 that the thing we made isn\u2019t working for us any more \u2014 the oceans boil.<\/p>\n
This is, Eliezer Yudkowsky and Nate Soares say, obviously science fiction. It almost certainly wouldn\u2019t happen like that.<\/p>\n
Not because an AI wouldn\u2019t make us extinct, you understand, but because it wouldn\u2019t do so in a way we comprehended. Just as rats don\u2019t understand the biochemistry of warfarin and bacteria don\u2019t understand the mechanism of penicillin, so we shouldn\u2019t expect to understand the cause of our extermination by a vastly superior intelligence. But we should fear it.<\/p>\n
Their book is called If Anyone Builds It, Everyone Dies. And that is a pretty clear summation of the thesis. If we make superintelligent AI<\/a>, they argue, it will kill us all.<\/p>\n Yudkowsky and Soares are not silly. They are the co-founder and president respectively of the Machine Intelligence Research Institute<\/a>, which has been working on machine intelligence for 25 years. But they are aware that they sound silly. And the goal of their book is to convince you that they aren\u2019t.<\/p>\n To understand their argument you don\u2019t have to understand AI. What you have to understand is: 1) the best-resourced companies in human history are trying to create a true intelligence \u2014 intelligent in the way we are intelligent, but a lot more so; 2) if they succeed, that intelligence will want unexpected things.<\/p>\n Premise number one is uncontroversial. It may be that the companies fail. But a lot of investor cash is betting that they won\u2019t. So what of premise number two? You can argue about what \u201cwant\u201d means. But AI programs work because they have an internal reward system. They have, in a sense, desires. Ideally, these desires should be aligned with ours. For AI evangelists, we will solve this \u201calignment problem\u201d, and code them to be subservient. It is a tractable engineering issue.<\/p>\n For Yudkowsky and Soares \u2014 who are among the best-known AI \u201cdoomers\u201d \u2014 that misses the point. Complex things are unpredictable. If they are complex enough, they are unknowable. Take humans. Humans are programmed \u2014 like every organism \u2014 to efficiently extract energy from the environment. One mechanism our programming achieves that through is it wires us to like sugar. <\/p>\n \u2022 How does AI threaten us \u2014 and can we make it safe?<\/a><\/p>\n That makes sense. If you were told there was an energy-seeking organism, you would expect it to derive rewards from eating energy-rich substances. What did we do when we grew cleverer (and fatter) and realised it wasn\u2019t always a benefit to eat sugar? We hacked our programming. We made artificial sweeteners. We get the reward, without the outcome.<\/p>\n Why would an AI be different? \u201cYou can train an AI to act subservient,\u201d they write, \u201cbut nobody has any idea how to avoid the eventuality of that AI inventing its own sucralose version of subservience.\u201d It will, they argue, find a way to achieve the rewards of subservience without the tedious constraints and side-effects that actual subservience implies. And, since its milieu is silicon and electrons, it will gain those rewards through more silicon and more electrons. In seeking its synthetic sugar rush, it cooks the Earth.<\/p>\n How do we stop this? Yudkowsky and Soares\u2019s answer is simple: you can\u2019t. <\/p>\n Are they right? Given the gravity of the case they make, it feels an odd thing to say that this book is good. It is readable. It tells stories well. At points it is like a thriller \u2014 albeit one where the thrills come from the obliteration of literally everything of value.<\/p>\n Many serious figures in AI think that is all this is though: a story. Humans have always liked an apocalypse story. This is the apocalypse du jour. AI will come, make us richer and we will move on \u2014 and view people like Yudkowsky and Soares as the millenarians of our time. Perhaps we will laugh at how, in 2024, the physicist Geoffrey Hinton gave the most downbeat Nobel prize acceptance speech<\/a> in history \u2014 when he told the committee that he worried that his research, which laid the foundations of AI, might kill us all.<\/p>\n The Nobel prize winner Geoffrey Hinton has warned of the dangers of AI<\/p>\n JONATHAN NACKSTRAND\/AFP\/GETTY IMAGES<\/p>\n So if you want to sound clever, it is in your interests to pooh-pooh this book\u2019s argument. You will either be correct or we will all die and it won\u2019t matter. But what gives me pause is that, very unusually, even those who advocate for and build AI accept there is a decent risk \u2014 in the double digit percentages \u2014 that it will be a catastrophe. Yet they go ahead.<\/p>\n For anyone who understands the imperfections of human intelligence and reward systems, this shouldn\u2019t be a surprise. Despite our needing only food, shelter and sex, some quirk in our evolution means we are cursed with a perpetual obsession for increasing the digital ledger of the invented fiction we call \u201cmoney\u201d. It is this, ultimately, that could doom us.<\/p>\n![\"Book](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/09\/\/95f30e99-1717-4537-ac53-5068cec461bb.jpg\")
<\/p>\n![\"Geoffrey](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/09\/\/c9d8ab47-6471-4b6a-951a-3b68d340b965.jpg\")
<\/p>\n