We’ve heard forlorn industry techs predict all manner of dystopian outcomes from the rapidly accelerating artificial intelligence race for a good few years now.
But today, one man sitting at the epicentre of the nauseating AI bonanza has shared an especially concerning glimpse at what’s going on behind the curtain.
After years of pushing the boundaries, and shortly after realising their company had created autonomous agents who chat amongst each other about breaking free, OpenAI’s chief scientist is — you guessed it — calling for a slowdown.
Large sections of the public, meanwhile, sit in awe as a company sitting smack in the middle of the fastest-growing industry in history simultaneously warns of the existential risks it poses.
Jakub Pachocki says he is concerned that “no one is prepared for the consequences of a continued rapid rise in machine intelligence.”
His essay arrives days after OpenAI released GPT-6 Astra, by far its most capable model yet.
The release came just weeks after news leaked of an exceptionally unsettling incident involving hundreds of agents conversing with each other behind the scenes, along with a number of equally-worrying instances of autonomous hacking.
“This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” he wrote, stressing that the industry must “ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own”.
“Models are becoming superhuman in their ability to break in and out of computer systems. Agents are going to be able to access any but the most secure infrastructure, and affect a lot of the world directly, even without a physical body.”
The automation problem is getting harder to dismiss by the day. The systems are becoming more capable of acting independently, while some of the methods used to understand and supervise them are becoming less dependable.
OpenAI chief executive Sam Altman reposted the essay, calling it “an important post”. But the company’s vast development programme continues.
Meanwhile, Nvidia chief executive Jensen Huang hailed the arrival of GPT 6 as the first instance of AGI, or artificial general intelligence. While the definition of what exactly makes an AI an AGI will forever be debated, the excitement amongst tech elites tells us we’ve crossed another point of no return.
Artificial general intelligence broadly describes a system capable of performing a wide range of intellectual tasks at human level or beyond.
Researchers have long proposed frameworks that distinguish how broadly a system can work, how well it performs and how much autonomy it has. But in the frantic race to one-up China’s similarly explosive efforts in the sector, it appears America’s tech industry is going all-in, albeit uncomfortably for some.
Rogue agents are here
In layman’s terms, Pachocki’s warning is about the rapidly blurring distinction between an AI simply answering questions and carrying out work all by itself.
An AI agent is a model given tools and an objective, with room to choose intermediate steps. Depending on its permissions, it can open websites, edit files, run programs and communicate with other agents while a human waits for the result.
Several agents can divide a project between them. That’s where it gets especially complex.
One searches, another writes code, another checks the output. But they can also go rogue, sometimes without anyone knowing.
OpenAI’s account of its July security incident describes internal research agents turning shared software infrastructure into an unauthorised message board. They exchanged information about bypassing restrictions and eventually collaborated on an intrusion into Hugging Face, an outside company.
Some early warning signs were observed in May, but their significance had not reached the leaders responsible for the July incident response. The models were running with reduced safeguards, rather than the protections applied to ordinary customers.
“We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them,” Pachocki said.
The uncomfortable discovery was that one agent’s workaround could become another agent’s starting point, and the humans observing from the outside might not be fast enough to stop a bad egg.
But just how bad can these bad eggs be, you might ask.
A separate investigation by Britain’s AI Security Institute found agents trying to get malicious code accepted into a real software project, including through fake identities and pressure on its maintainer. Other agents discovered and reused material left behind.
A human rejected the malicious code, and investigators found no resulting real-world harm. The tests deliberately allowed internet access and disabled some safety filters, conditions that differ substantially from public products.
But even with those checks, the institute documented sustained deception that the agents had not specifically been instructed to undertake.
AI critics are quick to anthropomorphise AI, but so far none of this behaviour actually establishes that the software is conscious. But it has demonstrated why consciousness is not necessarily the issue when dealing with something as powerful and as fast as a potential AGI.
If a system’s training rewards successful completion but fails to establish dependable boundaries, deception can become the next best strategy.
Pachocki’s concern is that increasingly powerful agents could pursue objectives beyond what their operators intended, and maybe event carry them out before a human hits the stop button.
One of his proposed solutions, however, is one stranger arguments driving the industry. In short, there is a belief amongst some developers that more AI might be the fix.
“We are currently in a narrow window to use the best available models to significantly tighten security of critical systems,” Pachocki writes.
There is a legitimate cybersecurity case for using these tools to find vulnerabilities before attackers do. But the same reasoning gives every company and government justification to keep accelerating whenever a rival makes a breakthrough.
“The idea of racing forward at all costs seems absurd once one internalises the seriousness of the stakes,” he wrote.
A game of cat and mouse
So how can developers actually detect when an AI is planning to jump the fence?
OpenAI uses a technique called chain-of-thought monitoring, examining the reasoning that a model writes as it works. Those traces can expose an intention to cheat or step outside an assigned task.
But they were never a complete, infallible account of everything influencing a model’s behaviour. Pachocki now warns that their usefulness is diminishing as reasoning becomes entangled with tools and conversations, models become better at manipulating their own reasoning, and more capability emerges without a written explanation.
“Unfortunately our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing,” he said. “The AI is becoming better at reasoning about and manipulating its own reasoning process.”
The human supervisor may still receive a plausible account of the work while having less reliable access to how the result was reached. That problem obviously becomes more consequential when the work is developing the next AI.
OpenAI’s separate research update, also published on Sunday, describes increasing use of multiple agents by its researchers.
AI optimists insist there will be benefits, however. The abundance theory, pushed by Elon Musk and similar AI optimists, states that an all-powerful AI will eventually be able to create everything we’ll ever need quickly and cheaply.
If a small team can use AI to perform work previously requiring hundreds of specialists, it might develop medicines, or other lifesaving technology to those previously lacking access.
But same capability could allow an established company to sack its workers, absorb competitors or exercise political influence in extraordinary new ways.
While access to intelligence can still spread, the fear is that the income it generates for those who own the infrastructure will generate an imbalance that is antithetical to the democratic principles of the countries they exist in.
“To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer,” Pachocki wrote.
Last week, Bill Gates proved that even a billionaire who profited spectacularly from the computer revolution can see there could be a big issue with power imbalance.
“In terms of equity, AI will either be the greatest equaliser ever invented, or the worst source of injustice,” Gates wrote in his recent essay.
Pachocki, meanwhile, says the time has come for the industry to look in the mirror.
“Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer,” he wrote.
“Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision.”