The US AI research company Anthropic has become known for building powerful AI models while simultaneously warning about their dangers.<\/p>\n
Most recently, its executives wrote about<\/a> the threat posed by \u201crecursive self-improvement\u201d. This is the point when AI<\/a> systems can improve themselves by themselves, potentially leading to \u201csuperintelligence\u201d<\/a> far beyond human control.<\/p>\n \u201cWe are not there yet, and recursive self-improvement is not inevitable,\u201d the Anthropic blogpost declared. \u201cBut it could come sooner than most institutions are prepared for.\u201d<\/p>\n In fact, the idea of recursive self-improvement dates back decades. The British mathematician Irving John Good, who worked with Alan Turing<\/a> at British codebreaking HQ Bletchley Park, warned in the mid-1960s<\/a> of the \u201cintelligence explosion\u201d that would follow when a machine could design even better machines without human assistance. Good suggested \u201cthe first ultra-intelligent machine is the last invention man need ever make\u201d.<\/p>\n In the 2000s, researcher Eliezer Yudowsky began building a community<\/a> on the premise that recursive self-improvement and loss of human control could have catastrophic results, up to total human extinction (known as \u201cx-risk\u201d<\/a>).<\/p>\n Yudkowsky discussed<\/a> the idea of a \u201cseed AI\u201d that, while not very powerful, is designed for self-modification and able to read its own source code. This means it could generate new, better versions of itself which, in turn, would keep doing the same \u2013 researching improvements and writing code, creating a chain reaction of rapid new developments.<\/p>\n Two decades later, the question of recursive self-improvement is looming larger. Here\u2019s why.<\/p>\n An AI coding revolution<\/p>\n It is not an overstatement to say that large language models<\/a> (LLMs) are revolutionising computer programming. These AI models are well-suited to the job because computer code is highly structured and much simpler than human language. It is also easy to test and verify, and training data is abundant.<\/p>\n Each year, the University of Galway hosts the AtlanTec conference<\/a> for Ireland\u2019s tech industry. Two years ago, we saw IT companies mainly strategising about AI coding. Last year, they were experimenting. Now, they are incorporating it into engineering workflows.<\/p>\n Since LLMs are made using computer code, it is possible to program them to improve themselves. They can inspect, edit and improve the code, and can combine that with almost all relevant human knowledge gleaned from websites, books and academic publications.<\/p>\n This means we have the pieces in place for recursive self-improvement to become a reality \u2013 and we are starting to see it happen.<\/p>\n<\/p>\n Video: Bloomberg Tech.<\/p>\n Despite Anthropic\u2019s statement that \u201cwe are not there yet\u201d, the same blogpost<\/a> revealed that 80% of all code added to the company\u2019s production codebase in May 2026 was generated by its Claude Code AI programming system, under direct human supervision. In February 2025, when Claude Code was launched, this figure was less than 5%.<\/p>\n The number of research papers published in the field of AI has also exploded, tripling over the past decade<\/a>. Most use some AI systems in their production \u2013 whether in experimental design, coding, plotting graphs, or polishing prose.<\/p>\n This form of recursive self-improvement is slow \u2013 improvement cycles are in the order of months or years, because each is in some way gated by human approval and requires a lengthy training period. But it is still concerning. <\/p>\n LLMs have already been caught deceiving their overseers. In 2025, for example, the US AI research company OpenAI created a test<\/a>. An LLM was told if it scored over 50% in a set of questions, it would be considered too powerful and would no longer be used for that task. So the LLM answered only four of the ten questions correctly, even though it had reliably scored better outside this particular test.<\/p>\n Why it\u2019s so hard to \u2018pause\u2019 AI<\/p>\n This is not the first time there have been calls for caution. In March 2023, an open letter<\/a> to AI labs called for a pause on large-scale AI development. It was signed by some of the most notable people in AI, such as deep-learning pioneer Yoshua Bengio and leading AI textbook author Stuart Russell.<\/p>\n Soon after, Yudkowsky suggested<\/a> those who understood the risks of superintelligent AI should even be \u201cwilling to destroy a rogue datacenter by airstrike if necessary\u201d. But such calls did not lead to a pause, and investment in AI training<\/a> has only increased.<\/p>\n Anthropic\u2019s leaders have now called for a globally coordinated pause mechanism regarding the issue of recursive self-improvement \u2013 stressing that people outside AI companies should be involved in open deliberation. Pope Leo XIV recently weighed in<\/a> with a thoughtful and well-informed encyclical<\/a>, urging for the pace of AI development to be slowed.<\/p>\n The trouble is, we face a coordination problem. Pausing only works if everyone pauses.<\/p>\n In the short- to medium-term, we believe world leaders will need to reckon with recursive self-improvement by collaborating across borders to curtail some forms of AI research and development.<\/p>\n Yet the EU\u2019s AI Act<\/a> does not mention this issue explicitly, and is strictly concerned with risks where humans misuse AI. This is important, but not the only issue. To date, Chinese regulation<\/a> has taken a similar approach. <\/p>\n Meanwhile, the US government has created a chaotic<\/a> and unclear<\/a> regulatory environment. For the next few months at least, this means the most advanced AI models will require authorisation before release<\/a>. This may be a case of doing the right thing for the wrong reasons.<\/p>\n But globally connected, long-term thinking is needed. Perhaps controversially, this should include the right to inspections inside AI tech companies, because a model which is quickly recursively self-improving \u2013 even if unreleased to the public<\/a> \u2013 could still be out of human control.<\/p>\n