It is fashionable in certain circles to dismiss the catastrophic risks of AI. One often hears that \u201cthe real experts\u201d who work on the technology every day are really not concerned at all; that only \u201cdoomers\u201d and \u201cluddites\u201d espouse a \u201cfringe\u201d view from a \u201cposition of ignorance\u201d; that all talk of potential catastrophe is just \u201cscience fiction\u201d.<\/p>\n
Fortunately, an open letter<\/a> has been published that lets us hear from the real experts who work on the technology every day, in their own words. And are they worried? Very.<\/p>\n The letter states: \u201cThere is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.\u201d It asks the US government to support \u201can international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development\u201d.<\/p>\n AI systems going \u201cbeyond our ability to understand or control\u201d would mean that we no longer have a say in whether we exist. AI systems escaping their confines and breaking into other companies\u2019 computers \u2013 something that has become an almost daily occurrence<\/a> \u2013 are an early warning.<\/p>\n To \u201cpace the frontier\u201d, as the letter asks, is to ensure that more advanced systems are developed and released only when it is demonstrably safe to do so. This requires a licensing process with strong verification of safety; registration of AI systems and the ability to monitor and terminate them when necessary; and enforceable international agreements. All of this takes time and is impossible to put in place once a real emergency happens.<\/p>\n The concerns expressed in the letter about the rapid acceleration of AI capabilities have been bubbling up for some months under the heading of \u201crecursive self-improvement\u201d or RSI \u2013 that is, AI systems contributing to their own improvement, which then enables even greater contributions to their own improvement, and so on.<\/p>\n This was the subject of the \u201cRecursive<\/a>\u201d conference in mid-May in San Francisco as well as Anthropic\u2019s RSI warning, \u201cWhen AI builds itself<\/a>\u201d, in early June. Broader concerns about the safety and societal impacts of AI have also motivated the formation of the Coalition of Concerned AI Staff<\/a>, which links together employees of all the major AI companies.<\/p>\n The letter is signed by (at the time of writing) 1,367 researchers and engineers at frontier AI labs \u2013 mainly OpenAI, Anthropic<\/a> and Google Deepmind. The list includes CEOs, co-founders, chief scientists and hundreds of well-known AI researchers, including many of my former students. These are the people who are driving the AI revolution while trying to fashion brakes and steering wheels at the same time. If these are not the experts, then there are none.<\/p>\n What makes the letter so powerful is not just the first-person authority of the signers but the inclusion of personal statements from almost a hundred of them. The statements paint a much more detailed picture of what the experts really think. At least three major themes emerge.<\/p>\n The first is the need to coordinate to escape an arms race towards the edge of cliff. Earlier this year at Davos, Dario Amodei, CEO of Anthropic, and Demis Hassabis, the CEO of Google<\/a> Deepmind, both suggested they wanted to halt AI development if others would agree. \u201cThe world is locked in a deadly race towards an intelligence explosion \u2026 To survive, we must coordinate to slow down the race,\u201d said one signer.<\/p>\n Another wrote: \u201cOnce this becomes common knowledge \u2013 once all of us in China and the US and at the various labs see that all the rest of us also think we need to slow this down \u2013 that\u2019s when coordination becomes possible.\u201d<\/p>\n The second common theme in the remarks is, as one signer put it, \u201cthe complete absence of credible plans for controlling \u2018superintelligent\u2019 AI systems\u201d. \u201cThese are the facts: The AI industry has the explicit goal of building things smarter than humans \u2026 None of us yet know how to make sure these things stay under human control \u2026 This is, objectively, an insane and suicidal thing to do,\u201d wrote another.<\/p>\n Not surprisingly, these concerns lead to the third theme: real fear. \u201cI currently feel quite afraid of all paths I see that don\u2019t include a near-future negotiated slowdown.\u201d \u201cThere is \u2026 a very dark future ahead of us if we don\u2019t get this right.\u201d And: \u201cI want a world where my kids can have a full life not a world where they are wiped out.\u201d<\/p>\n These are not greedy employees trying to hype their stock valuations. They say the same things in private conversations, closed-door meetings and off-the-record chat groups.<\/p>\n In the face of such real and well-founded fears, surely these trillion-dollar companies are devoting massive resources to ensure they don\u2019t end our, and their own, existence? Wrong.<\/p>\n One researcher, quoted in a recent New Yorker article<\/a> on the OpenAI systems that broke into Hugging Face, said: \u201cIf people actually knew what the safety culture looks like, even in the most safety-minded labs, then I think people would be genuinely much more freaked out.\u201d<\/p>\n According to one senior safety researcher at Anthropic, their best bet for avoiding an irreversible loss of control works like this: as soon as the superintelligent machine comes into existence, but before it\u2019s had a chance to take over the world, we ask it how we can stop it from taking over the world, and we do what it says.<\/p>\n This plan doesn\u2019t fill me with confidence, and the Anthropic researcher gives it at best a 50% chance of working. It\u2019s the same plan that IJ Good mentions in his famous 1965 paper<\/a> on the intelligence explosion: \u201cThe first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.\u201d I rather think that Good meant this remark sarcastically.<\/p>\n The most powerful of all the personal signing statements is also the shortest: \u201cThink of all we love.\u201d<\/p>\n