Want to bookmark your favourite articles and stories to read or reference later? Start your Independent Membership today.
Already a member?
Log in
It is, perhaps, the ultimate promise of artificial intelligence: AI systems that can improve themselves, going round and round as they reach a virtuous cycle until they reach artificial general intelligence, or AGI, becoming better than humans at most tasks and provide us with whole new ways of computing. It is also, some warn, the ultimate threat, and could lead with all of us being wiped out.
Both are views of recursive self-improvement, or RSI, a long-discussed belief among AI experts that some claim might finally be arriving. Nobody really knows for sure whether it could happen, and they are even less sure about whether it should.
It is not, as far as we know, happening yet. Both OpenAI and Anthropic have said that the technology is not ready yet – though the latter has said that AI tools are already speeding up AI development, pointing to numbers that suggest Anthropic engineers are producing eight times more code than they did before 2025.
AI-powered humanoid robots at the 2nd World Robot Games at National Speed Skating Oval on 26 August, 2026 in Beijing, China (Kevin Frayer/Getty Images)
Indeed, nobody really even knows if it will ever happen. For RSI to be possible, we will need even more powerful AI systems, and while frontier labs suggest they are arriving quickly, it is in their interests to be positive.But whether it actually arrives, all experts suggest that it may be close enough that we should start thinking about both its promises and its threats. And there are plenty of both.
The promise is perhaps the most obvious. AI being able to build itself would also mean that it would become far more powerful, far more quickly – and thereby lead to all of the kinds of developments in healthcare, science, technology and so much more besides that its evangelists have been promising for so long.
Another largely unspoken reason for the excitement about recursive self-improvement is desperation: AI developers are urgently looking for ways to improve their models, and running out. Artificial intelligence systems are, generally, as good as the data they are trained on – and so those making them are constantly on the hunt for more data to ingest.
But as AI labs look for new ways to improve their models, they are forced to scoop up more and more of that data, inevitably meaning that they can be less choosy and thereby include more lower-quality material. Eventually, at some period that might not be so far away, that data might run out, and the improvement would slow or stop.
(Getty Images/iStockphoto)
If that whole process could be automated, then data and humans, which are currently required both to build and evaluate the automated systems that power AI, will no longer be the bottleneck holding back development. Instead, the promise of RSI is that the development of AI will only be limited by how much computing power is available, and AI companies are working to provide more of that each day.
But that too is the reason for the threat. If humans are removed from the development, and are less able to monitor it, then all of the dangers of AI only become amplified.
If AI systems are improving themselves, for instance, then we will need to think about what “improvement” means, and give the results of that thinking to the systems so they know the direction they can point. But that could only lead to “reward hacking”, where automated systems find unexpected, unethical or outright dangerous ways to get to their goals.
The warnings about that kind of danger are only becoming more real by the day. In the various cases of major AI tools hacking other systems, for instance – such as an incident revealed this week where Australia said than an OpenAI agent had launched a cyber attack on the government – it has been the result of tools finding a more efficient way of satisfying their goals, and it leading to behaviour that nobody had actually asked for or wanted.
Some of this danger could be ameliorated by AI itself, the companies building it have suggested. In a blog post this week, OpenAI said that it was working towards “building an automated AI researcher”, that it could with work on the “alignment problem”, or the difficulty of ensuring that AI behaves in line with what humans want and think automated systems should do.
“Automated research could also help us substantially improve alignment and build defenses against increasingly capable AI—an automated AI researcher can also be an automated AI safety researcher,” the company wrote. “More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures.”
But if nobody knows for sure whether or how RSI will happen, then it is even less clear whether it will actually make us safer. This is one of the many reasons that companies from OpenAI to Anthropic have called for a slowdown in development, so that they can spend more time understanding the threats and responding to them.
But those calls for slowdowns – which have now come from just about every major AI lab – have been snarled up in concerns about rogue AI labs racing ahead, or countries such as the US losing their global dominance over the technology. The promise of further development, and of that development being powered by AI, may prove simply too alluring – and the only way we find out whether the cycle of recursive self-improvement happens, and is a vicious or virtuous one, might be in practice.