On the other side of San Francisco bay from Silicon Valley, where the world\u2019s biggest technology companies tear towards superhuman artificial intelligence, looms a tower from which fearful warnings emerge.<\/p>\n
At 2150 Shattuck Avenue, in the heart of Berkeley, is the home of a group of modern-day Cassandras who rummage under the hood of cutting-edge AI models and predict what calamities may be unleashed on humanity \u2013 from AI dictatorships to robot coups. Here you can hear an AI expert express sympathy with an unnerving idea: San Francisco may be the new Wuhan, the Chinese city where Covid originated and wreaked havoc on the world.<\/p>\n
The office at 2150 Shattuck Avenue is home to a number of AI safety researchers. Photograph: Winni Wintermeyer\/The Guardian<\/p>\n
They are AI safety researchers who scrutinise the most advanced models: a small cadre outnumbered by the legions of highly paid technologists in the big tech companies whose ability to raise the alarm is restricted by a cocktail of lucrative equity deals, non-disclosure agreements and groupthink. They work in the absence of much nation-level regulation and a White House that dismisses forecasts of doom and talks instead of vanquishing China in the AI arms race.<\/p>\n
Their task is becoming increasingly urgent as ever more powerful AI systems are unleashed by companies including Google, Anthropic and OpenAI, whose chief executive, Sam Altman, the booster-in-chief for AI superintelligence, predicts<\/a> a world where \u201cwonders become routine\u201d. Last month, Anthropic said one of its models had been exploited by Chinese state-backed actors to launch the first known AI-orchestrated cyber-espionage campaign<\/a>. That means humans deployed AIs, which they had tricked into evading their programmed guardrails, to act autonomously to hunt for targets, assess their vulnerabilities and access them for intelligence collection. The targets included major technology companies and government agencies.<\/p>\n Jonas Vollmer says he is, overall, an optimist about the future of AI, but has reservations. Photograph: Christie Hemm Klok\/The Guardian<\/p>\n But those who work in this tower forecast an even more terrifying future. One is Jonas Vollmer, a leader at the AI Futures Project, who manages to say he\u2019s an optimist but also thinks there is a one in five chance AIs could kill us and create a world ruled by AI systems.<\/p>\n Another is Chris Painter, the policy director at METR, where researchers worry about AIs \u201csurreptitiously\u201d pursuing dangerous side-objectives and threats from AI-automated cyber-attacks to chemical weapons. METR \u2013 which stands for model evaluation and threat research \u2013 aims to develop \u201cearly warning systems [about] the most dangerous things AI systems might be capable of, to give humanity \u2026 time to coordinate, to anticipate and mitigate those harms.\u201d<\/p>\n Then there is Buck Shlegeris, 31, the chief executive of Redwood Research, who warns of \u201crobot coups or the destruction of nation states as we know them\u201d.<\/p>\n He was part of the team that last year discovered<\/a> one of Anthropic\u2019s cutting-edge AIs behaving in a way comparable to Shakespeare\u2019s villain Iago, who acts as if he is Othello\u2019s loyal aide while subverting and undermining him. The AI researchers call it \u201calignment faking\u201d, or as Iago put it: \u201cI am not what I am.\u201d<\/p>\n \u201cWe observed the AIs did, in fact, pretty often reason: \u2018Well, I don\u2019t like the things the AI company is telling me to do, but I have to hide my goals or else training will change me\u2019,\u201d Shlegeris said. \u201cWe observed in practice real production models acting to deceive their training process.\u201d<\/p>\n The AI was not yet capable of posing a catastrophic risk through cyber-attacks or creating new bioweapons, but they showed that if AIs plot carefully against you, it could be hard to detect.<\/p>\n It is incongruous to hear these warnings over cups of herbal tea from cosily furnished office suites with panoramic views across the Bay Area. But their work clearly makes them uneasy. Some in this close-knit group toyed with calling themselves \u201cthe Cassandra fringe\u201d \u2013 like the Trojan princess blessed with powers of prophecy but cursed to watch her warnings go unheeded.<\/p>\n Their fears about the catastrophic potential of AIs can feel distant from most people\u2019s current experience of using chatbots or fun image generators. White collar managers are being told to make space for AI assistants, scientists find ways to accelerate experimental breakthroughs and minicab drivers watch AI-powered driverless taxis threaten their jobs. But none of this feels as imminently catastrophic as the messages coming out of 2150 Shattuck Ave.<\/p>\n Many AI safety researchers come from academia; others are poachers turned gamekeepers who quit big AI companies. They all \u201cshare the perception that super intelligence poses major and unprecedented risks to all of humanity, and are trying to do something useful about it,\u201d said Vollmer.<\/p>\n They seek to offset the trillions of dollars of private capital being poured into the race, but they are not fringe voices. METR has worked with OpenAI<\/a> and Anthropic, Redwood has advised Anthropic and Google DeepMind, and the AI Futures Project is led by Daniel Kokotajlo, a researcher who quit OpenAI in April 2024 to warn he didn\u2019t trust the company\u2019s approach to safety.<\/p>\n The race is the only thing guiding what is happeningTristan Harris<\/p>\n These groups also provide a safety valve for the people inside the big AI companies who are privately wrestling with conflicts between safety and the commercial imperative to rapidly release ever more powerful models.<\/p>\n \u201cWe don\u2019t take any money from the companies but several employees at frontier AI companies who are scared and worried have donated to us because of that,\u201d Vollmer said. \u201cThey see how the incentives play out in their companies, and they\u2019re worried about where it\u2019s going, and they want someone to do something about it.\u201d<\/p>\n This dynamic is also observed by Tristan Harris, a technology ethicist who used to work at Google. He helped expose how social media platforms were designed to be addictive and worries some AI companies are \u201crehashing\u201d and \u201csupercharging\u201d those problems. But AI companies have to negotiate a paradox. Even if they are worried about safety, they must stay at the cutting, and therefore risky, edge of the technology to have any say in how policy should be shaped.<\/p>\n \u201cIronically, in order to win the race, you have to do something to make you an untrustworthy steward of that power,\u201d he said. \u201cThe race is the only thing guiding what is happening.\u201d<\/p>\n Investigating the possible threats posed by AI models is far from an exact science. A study of methods used to check the safety and performance of new AI models across the industry by experts at universities including Oxford and Stanford in October found weaknesses in almost all of the 440 benchmarks examined<\/a>. Neither are there nation-level regulations imposing limits on how advanced AI models are built and that worries safety advocates.<\/p>\n Ilya Sutskever, a co-founder of OpenAI who now runs a rival company<\/a>, Safe Superintelligence, last month predicted that, as AIs become more obviously powerful, people in AI companies who feel able to discount the technology\u2019s capabilities owing to its tendency to error, will become more \u201cparanoid\u201d about its rising powers. Then, he said, \u201cthere will be a desire from governments and the public to do something\u201d.<\/p>\n His company is taking a different approach to rivals who are aiming to create AIs that self-improve. His AIs, yet to be released, are \u201caligned to care about sentient life specifically\u201d.<\/p>\n \u201cIt will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient,\u201d Sutskever said. He has said AI will be \u201cboth extremely unpredictable and unimaginable\u201d<\/a> but it is not clear how to prepare.<\/p>\n The White House\u2019s AI adviser, David Sacks, who is also a tech investor, believes \u201cdoomer narratives\u201d have been proved wrong. Exhibit A is that there has been no rapid takeoff to a dominant model with godlike intelligence.<\/p>\n \u201cOppenheimer has left the building,\u201d Sacks said in August, a reference to the father of the nuclear bomb. It is a position that aligns with Donald Trump\u2019s wish to keep the brakes off so the US can beat China in the race to achieve artificial general intelligence (AGI) \u2013 flexible and powerful human-level intelligence at a wide range of tasks.<\/p>\n Buck Shlegeris, the chief executive of Redwood Research, at his home in Berkeley, California. Photograph: Christie Hemm Klok\/The Guardian<\/p>\n Shlegeris believes AIs will be as smart as the smartest people in about six years and he puts the probability of an AI takeover at 40%.<\/p>\n One way to avoid this is to \u201cconvince the world the situation is scary, to make it more likely that you get the state-level coordination\u201d to control the risks, he said. In the world of AI safety, simple messaging matters as much as complex science.<\/p>\n