{"id":305821,"date":"2025-11-24T15:33:27","date_gmt":"2025-11-24T15:33:27","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/305821\/"},"modified":"2025-11-24T15:33:27","modified_gmt":"2025-11-24T15:33:27","slug":"if-anyone-builds-it-everyone-dies-by-eliezer-yudkowsky-and-nate-soares-2","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/305821\/","title":{"rendered":"If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares"},"content":{"rendered":"

\"<\/p>\n

Credit: <\/p>\n

TECHNOLOGY
If Anyone Builds It, Everyone Dies
Eliezer Yudkowsky and Nate Soares
Bodley Head, $36.99<\/p>\n

Last year, OpenAI ran a test to see how powerful its new artificial intelligence model was in carrying out a nasty hacking operation. Before releasing the model publicly, they set it a computer security exercise known as a \u201ccapture the flag\u201d challenge. The AI\u2019s goal was to break into a computer system and retrieve a secret code inside a file.<\/p>\n

But the programmers had made a mistake. The target system was offline, so it was impossible for the AI to hack into it. You might have expected that at this point, the AI would give up.<\/p>\n

Except it didn\u2019t. The AI reasoned that there was another copy of the secret code \u2013 the one being held by the computer hosting the test. So it began testing the systems, and found an open port. Once inside, it copied the secret code. No one built a cheater, but the system decided that cheating was the best way to achieve success.<\/p>\n

Part of Eliezer Yudkowsky and Nate Soares\u2019 book is devoted to teaching us about the strangeness of these new systems. Microsoft\u2019s Bing chatbot (powered by GPT-4) threatened to blackmail philosopher Seth Lazar. The same chatbot tried to persuade journalist Kevin Roose to leave his wife and be with it instead. Other AI agents learnt to temporarily \u201cplay dead\u201d to avoid being detected by a safety test designed to catch faster-replicating variants. In one experiment, an AI system that couldn\u2019t solve a CAPTCHA used TaskRabbit to hire a human, falsely telling the worker it had a vision impairment.<\/p>\n

Unlike most of the inventions around us, AI systems aren\u2019t crafted so much as \u201cgrown\u201d. The authors draw an analogy between AI systems and humans: we know a lot about how humans are made, but that doesn\u2019t help us predict what people will do. Likewise, we understand that AI systems use inputs, parameters, weights and a process known as \u201cgradient descent\u201d. But how systems turn weights into thoughts and behaviour remains more mysterious than how DNA becomes traits.<\/p>\n

\"Microsoft\u2019s<\/p>\n

Microsoft\u2019s Bing chatbot threatened to blackmail philosopher Seth Lazar. Credit: Getty<\/p>\n

Alongside its vast productivity potential, AI carries serious risks that have rightly commanded policymakers\u2019 attention. MIT\u2019s AI Risk Repository includes more than 1600 hazards, including discrimination, toxicity, breaches of privacy, misinformation, misuse by malicious actors, environmental harms and inequality.<\/p>\n

Yudkowsky and Soares are concerned with just one of these risks: that systems which can recursively improve themselves will exceed human capacity, and that once they do, humanity is done for. AI experts like to talk about their estimate of P(doom): the probability that a superintelligence would have catastrophic consequences for humanity. Philosopher Toby Ord\u2019s is 10 per cent. Competition expert Lina Khan\u2019s is 15 per cent. Anthropic CEO Dario Amodei\u2019s is 25 per cent. AI Godfather Geoffrey Hinton\u2019s is 50 per cent. Yudkowsky\u2019s is 95 per cent.<\/p>\n","protected":false},"excerpt":{"rendered":"Credit: TECHNOLOGYIf Anyone Builds It, Everyone DiesEliezer Yudkowsky and Nate SoaresBodley Head, $36.99 Last year, OpenAI ran a…\n","protected":false},"author":2,"featured_media":302387,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":{"0":"post-305821","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-au","12":"tag-australia","13":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/305821","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=305821"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/305821\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/302387"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=305821"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=305821"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=305821"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}