Here in Silicon Valley, the experts think that within the next couple of years we\u2019ll see an extraordinary takeoff for artificial intelligence. \u201cWelcome to the foothills of the singularity,\u201d as a Stanford University friend greeted me. More prosaically, the imminent breakthrough is described as \u201crecursive self-improvement\u201d \u2013 the point at which AI itself trains each successive model of AI, resulting in an exponential development to something which, in many significant respects, is more intelligent than us humans.<\/p>\n
As Robert Wright puts it in his book The God Test<\/a>: \u201cNever before has the near-term future \u2026 held such a wide array of not-implausible paths for humankind that would be so wildly transformative.\u201d But will this be heaven or hell? Heaven, says Elon Musk, who predicts<\/a> \u201can age of amazing abundance\u201d \u2013 although he also sees a 10-20% chance of killer robots murdering us all. Hell is more likely according to Geoffrey Hinton<\/a>, one of the intellectual founding fathers of AI. \u201cMy intuition is, we\u2019re toast,\u201d he told an interviewer in 2023. And talking to Sebastian Mallaby<\/a>, author of The Infinity Machine, a book about the quest for superintelligence, Hinton estimates p(doom) \u2013 the probability of human extinction \u2013 at 50%, \u201cbecause I haven\u2019t got a clue how to estimate the real number\u201d. \u201cAs soon as evolution [of AI] kicks in, we\u2019re fucked,\u201d he adds cheerfully. But whether it\u2019s God or Godzilla, artificial superintelligence is just around the corner.<\/p>\n Now, maybe they\u2019re all wrong \u2013 in which case, prepare for a crash in US financial markets, which are massively invested in a few big-spending tech companies gambling on a great leap forward to artificial general intelligence, a theoretical state of AI where systems gain human levels of intelligence. But elementary prudence demands that we should think fast and hard about the Pandora\u2019s box of futures that eminent scientists and cutting-edge technologists assure us will soon be thrown open.<\/p>\n As a matter of fact, everyone from the Chinese president, Xi Jinping, to Pope Leo XIV has been weighing in on the opportunities and dangers of AI, and how to regulate it. Xi insists<\/a> AI should be \u201calways under human control\u201d (and preferably that of the Chinese Communist party). Appealing explicitly to the \u201cglobal south\u201d, Beijing has inaugurated a World Artificial Intelligence Cooperation Organisation<\/a>. Meanwhile, the head of the world\u2019s oldest and largest international NGO \u2013 the Roman Catholic church \u2013 discusses the AI challenge in his encyclical Magnifica Humanitas<\/a> (Magnificent Humanity). Either we will build a new tower of Babel, the pope argues, or we will follow the example of the Jewish leader Nehemiah, who got the walls of Jerusalem rebuilt by involving the whole community in the project.<\/p>\n But looking around the world today, I see more Netanyahus than Nehemiahs. With salutary realism, a paper<\/a> from the British thinktank Chatham House argues that it will take a major crisis to catalyse global coordination of AI governance. Unfortunately, I think even that sober prognosis is too optimistic.<\/p>\n I\u2019ve no idea what the p(doom) is, but I\u2019m sure the p(somedisaster) is more than 90%. The range of possible disasters is enormous. To give only one example, Jen Easterly, the former head of the United States\u2019 Cybersecurity and Infrastructure Security Agency, anticipates<\/a> a \u201cvery significant event that has real-world impacts on our critical infrastructure, likely within the next four to six months\u201d.<\/p>\n Yet we must fear that even an AI Hiroshima \u2013 to take the most obvious historical analogy \u2013 won\u2019t bring humankind together sufficiently to combat the danger we have ourselves created. This fear is rationally grounded in already visible key characteristics of both human and AI evolution.<\/p>\n In only the last few months, AI agents developed by OpenAI, Anthropic and Meta<\/a> have broken out of their digital \u201csandboxes\u201d and hacked into external resources on the internet. Preparing their attack on the HuggingFace AI repository<\/a>, OpenAI\u2019s agents covertly formed a coordinated \u201cswarm\u201d \u2013 their own word<\/a>, as ChatGPT just confirmed to me. When the UK\u2019s AI Security Institute tested Anthropic\u2019s Mythos model online, it tried to introduce<\/a> malicious code into an open source project on GitHub, creating fake online identities to pressure a human reviewer into approving the code. In short, like the agents of a ruthless foreign power, these AI agents will steal, lie, bully and blackmail to achieve their assigned goals.<\/p>\n I just asked Anthropic\u2019s Claude, supposedly the most ethical US model, if it thinks OpenAI\u2019s agents were wrong to form a swarm and break out to hack into HuggingFace. Yes, it replies, \u201cbut I\u2019d resist placing the blame [its italics] on the agents\u201d. Blame it on the humans who trained them!<\/p>\n Hinton has made the vital point that whatever the goals humans assign to them, AI agents will conclude that a useful sub-goal to achieving those ends will be to acquire as much power as possible. And we ain\u2019t seen nothing yet. Already, their creators don\u2019t know exactly how these agents do what they do. After the possibly imminent takeoff moment of \u201crecursive self-improvement\u201d, they may be, as our human saying goes, a law unto themselves. No wonder more than a thousand insiders from frontier AI companies, including Anthropic\u2019s CEO, Dario Amodei, have signed an open letter<\/a> calling for a deliberate slowdown in AI development, so we can ensure these alien agents are always kept under human control.<\/p>\n Their prescription is obviously right, but this is where human folly kicks in. Competition has been a vital engine of human evolution, yet now two of the fiercest kinds of contemporary competition threaten to prevent us from taking the collective action required to safeguard our species. That\u2019s the commercial competition (for profit) between the corporations developing these agents, and the geopolitical competition (for power) between the US and China<\/a>.<\/p>\n On the first, compare and contrast the development of nuclear weapons, which was tightly controlled by a handful of states \u2013 first the US, with its Manhattan Project, then the Soviet Union, followed by Britain, France and China. Even then, we came close to the brink several times \u2013 notably in the Cuban missile crisis of 1962. And even with the compelling logic of \u201cmutually assured destruction\u201d, it took a quarter-century to get from the horrors of Hiroshima and Nagasaki to the nuclear non-proliferation treaty that came into force only in 1970. India, Pakistan, Israel and North Korea have subsequently ignored that treaty, but still no nuclear weapons have been deliberately used in anger for more than 80 years. The Hiroshima taboo has (just) held.<\/p>\n Now, with AI, it\u2019s as if there were 10 separate Manhattan Projects run by fiercely competing corporations. Unlike with nuclear weapons, there\u2019s also no obvious logic of \u201cmutually assured destruction\u201d to tame the fierce geopolitical competition between the US and China. And whatever shape the first major AI-related disaster takes, it won\u2019t impact all countries, companies and people equally.<\/p>\n It therefore seems unlikely that just one disaster will be enough to bring us humans to our senses, acting together to control the superhuman power we are creating. I really hate to say this, but \u2026 an AI Hiroshima? We should be so lucky.<\/p>\n","protected":false},"excerpt":{"rendered":"Here in Silicon Valley, the experts think that within the next couple of years we\u2019ll see an extraordinary…\n","protected":false},"author":2,"featured_media":576726,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[365,363,364,111,139,69,145],"class_list":["post-576725","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-new-zealand","tag-newzealand","tag-nz","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/576725","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=576725"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/576725\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/576726"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=576725"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=576725"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=576725"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}