{"id":785426,"date":"2026-09-28T16:08:38","date_gmt":"2026-09-28T16:08:38","guid":{"rendered":"https:\/\/www.newsbeep.com\/uk\/785426\/"},"modified":"2026-09-28T16:08:38","modified_gmt":"2026-09-28T16:08:38","slug":"wheres-the-intelligence-explosion-by-ramez-naam","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/uk\/785426\/","title":{"rendered":"Where\u2019s the \u201cintelligence explosion\u201d? – by Ramez Naam"},"content":{"rendered":"

\"\"<\/a>Art by GPT-6<\/p>\n

One of my fundamental beliefs about the world is that Ramez Naam<\/a> ought to blog more. Ramez is one of the world\u2019s greatest futurists \u2014 he predicted the solar<\/a> and battery revolutions long before these were widely understood. If you were reading Ramez in 2011, you were able to understand the future of both energy technology and climate change, long before other people did. His earlier book More than Human is still a great guide to the kind of biological enhancements that AI might make possible. Ramez is also an excellent science fiction author, having written a trilogy of novels<\/a> in which nanotechnological telepathy is distributed as a party drug (I\u2019m not sure if he actually expects that to happen, but it\u2019s a very cool idea). <\/p>\n

Unfortunately, although he does have a Substack<\/a> (which you should absolutely follow), Ramez does not blog regularly. However, after having a lengthy private debate with him about Recursive Self-Improvement, I was able to prevail upon him to write up his thoughts for my blog. <\/p>\n

To say that RSI is a big deal in the AI world would be a colossal understatement. Among AI researchers, entrepreneurs, and AI safety people, there\u2019s a widespread belief that as AI gets better at improving itself, there will be a \u201cfast takeoff\u201d or \u201cFOOM\u201d, in which AI\u2019s capabilities \u201ctake off\u201d and create a technological Singularity<\/a>. This event is a staple of science fiction, including works by my favorite sci-fi author, Vernor Vinge<\/a>. <\/p>\n

A lot of people in the industry believe that this moment is now close at hand, and are racing toward that prize:<\/p>\n

\"X<\/p>\n

Tanay Jaipuria@tanayj<\/p>\n

Noam Brown says that recursive self-improvement \/ the ability of AI models to do AI research is the top priority for OpenAI by a wide margin. <\/p>\n

OpenAI ranks it above making models that sell.<\/p>\n

(via @theinformation podcast)<\/p>\n

12:24 AM \u00b7 Sep 27, 2026 \u00b7 4.01K Views<\/p>\n

4 Replies \u00b7 2 Reposts \u00b7 44 Likes<\/p>\n

<\/a><\/p>\n

But Ramez \u2014 normally among the most wide-eyed of techno-optimists \u2014 is highly skeptical that we\u2019ll see anything like the \u201cFOOM\u201d of Vernor Vinge novels. In this lengthy, well-researched post, he explains his skepticism. <\/p>\n

Personally, I\u2019m agnostic. Ramez\u2019s case necessarily rests on a lot of assumptions; although it\u2019s cogently laid out, I think the real answer is that we\u2019ll just have to wait and see whether the Singularity arrives. But even more fundamentally, I don\u2019t know how much this debate matters in the practical sense \u2014 even without the kind of Singularity depicted in sci-fi novels, AI capabilities are improving so rapidly that they\u2019re already superhuman in many respects, and soon will probably be strongly superhuman in most or all dimensions. The AI of 2040 is going to look godlike, whether or not it explodes into an actual god in 2027. <\/p>\n

Still, it\u2019s a very interesting argument, and Ramez\u2019s thoughts on the future of technology are always worth listening to. <\/p>\n

AI is already helping improve itself. The question is whether even fully autonomous recursive self-improvement (RSI) would cause a runaway intelligence explosion.<\/p>\n

The theory is that each generation of AI could build a better successor, faster than the last generation did. That could lead to a \u201cfast takeoff,\u201d with capabilities surging to artificial superintelligence (ASI) in a year, months, or even days.<\/p>\n

Here\u2019s my take: Given our best current data, the AI self-improvement loop would need to be roughly 5\u201310\u00d7 stronger to sustain itself, let alone run away. I\u2019ll explain this math in section 8<\/a>. I expect incredibly rapid AI progress by the standards of nearly any other technology. But the evidence we have doesn\u2019t suggest a sudden explosion to incomprehensible superintelligence anytime soon.<\/p>\n

I could be wrong. Forecasters have repeatedly underestimated AI progress<\/a>! I could well be next. One thing that\u2019s clear is that we need better data<\/a>. For now, let\u2019s work with what we can measure, and stay open to breakthroughs that could change the picture.<\/p>\n

\"\"<\/a><\/p>\n

Figure 1. How strong is the self-improvement loop? Model<\/a>.<\/p>\n

Jump to the conclusion<\/a>.<\/p>\n

Here\u2019s the case, with links to each part:<\/p>\n

Narrow<\/a> Superintelligence Is Here Today<\/a><\/p>\n

Real-World Research Is Harder<\/a><\/p>\n

Impressive AI Numbers \u2192 Sharp Diminishing Returns<\/a><\/p>\n

We\u2019re Not Seeing Signs of Acceleration<\/a><\/p>\n

Keeping Up the Pace Takes Exponentially More Resources<\/a><\/p>\n

Better AI May Be Needed Just to Maintain the Pace<\/a><\/p>\n

Progress Gets Harder; Ideas Get Harder to Find<\/a><\/p>\n

The Current Feedback Loop Doesn\u2019t Look Strong Enough<\/a><\/p>\n

OpenAI\u2019s Data Shows How Weak the Loop Is<\/a><\/p>\n

What Could Accelerate Progress?<\/a><\/p>\n

We Need More Data to Track This Well<\/a><\/p>\n

Key charts: The feedback loop<\/a> \u00b7 Measured vs. forecast progress<\/a> \u00b7 Diminishing returns<\/a><\/p>\n

People use \u201crecursive self-improvement\u201d to mean everything from AI boosting the productivity of human researchers to AI bootstrapping itself to incomprehensible intelligence. Here\u2019s my taxonomy: productivity gains (Type 1), increasing autonomy while still facing diminishing returns (Types 2\u20134), and a runaway loop to superintelligence if we can ever find accelerating returns (Type 5).<\/p>\n

\"\"<\/a><\/p>\n

Figure 2. Five types of AI self-improvement.<\/p>\n

We\u2019ve made real progress on Types 1 and 2: AI helps both researchers and engineers inside of AI companies, and powerful models can train and improve smaller ones. We haven\u2019t yet seen clear evidence for Type 3 (though Alibaba just made some strong claims<\/a>) and certainly not for Type 4. I do expect autonomous self-improvement to arrive at some point. I\u2019m skeptical that it leads to Type 5 – runaway super-intelligence – without a major conceptual breakthrough.<\/p>\n

There are plenty of other definitions of RSI, which can be a bit confusing. Weco\u2019s four levels of RSI<\/a> are close to mine. For a broader tour of all the things people mean when they say \u2018RSI\u2019, read Tom Cunningham\u2019s comprehensive guide<\/a>.<\/p>\n

I do expect narrow superintelligence in highly verifiable domains. Think chess, Go, formal math, parts of computer science and coding. Highly verifiable domains are largely formal and structured types of work where machines can generate unlimited training data, with perfect or near-perfect verification of correct vs incorrect, and do so entirely in software without waiting on the physical world or humans. That\u2019s an ideal setting for AI learning.<\/p>\n

\"\"<\/a><\/p>\n

Figure 3. What makes a domain highly verifiable?<\/p>\n

In fact, we already have narrow superintelligence in game plang. We\u2019re seeing it happen now in the most formal parts of math, in particular in proofs and in finding counter-examples that disprove major conjectures. For example, OpenAI recently reported an AI-generated proof resolving the Navier\u2013Stokes existence and smoothness problem<\/a>. Parts of software development are also extremely verifiable, while others are a bit less crisp (such as understanding what humans want).<\/p>\n

That isn\u2019t the same as broad superintelligence. Even our most powerful models need far more training data than humans, struggle to learn reliably from ongoing experience, and fail in surprising ways on tasks people find straightforward. Superhuman math doesn\u2019t automatically mean superhuman judgment everywhere else.<\/p>\n

Back to contents<\/a><\/p>\n

Benchmarks and forecasts suggest that AI models should reliably succeed at coding tasks that take humans hours, without human help. The real world is messier. OpenAI\u2019s internal data shows much shorter stretches of autonomous work on research tasks.<\/p>\n

In its Research Acceleration \/ RSI report<\/a>, OpenAI showed how often its models completed tasks with and without human help, grouped by how long a human would need to do the work.<\/p>\n

\"\"<\/a><\/p>\n

Figure 4. OpenAI\u2019s internal research tasks. Source<\/a>.<\/p>\n

Even on tasks that would take a human less than 15 minutes, OpenAI\u2019s models succeeded without human intervention only 86% of the time. The estimated task length at 80% success was roughly 15 minutes over the first seven months of the year. July\u2019s results were similar to the whole period average.<\/p>\n

Fully autonomous RSI would require an AI to string together a great many research tasks reliably, stretching out over complex tasks that humans need weeks or months to accomplish. OpenAI\u2019s data suggests that we aren\u2019t close.<\/p>\n

Anthropic also released a graph<\/a> showing how Claude accelerates AI research. It shows that internal AI models collaborate on or even lead more than 90% of R&D tasks. That\u2019s objectively impressive. At the same time, the graph reports zero cases of AI autonomously completing AI R&D tasks.<\/p>\n

\"\"<\/a><\/p>\n

Figure 5. Claude\u2019s role in internal AI R&D. Source<\/a>.<\/p>\n

These are incredible tools. But they still need skilled people to set direction and get them back on track.<\/p>\n

For years, METR<\/a> has been publishing a chart showing what length of coding task (measured in human hours to complete) best-in-class AI models can achieve. It\u2019s been called the most important graph in AI<\/a>. METR\u2019s Mythos Preview evaluation estimated that the model could succeed at 80% of coding tasks that took humans three hours.<\/p>\n

\"\"<\/a><\/p>\n

Figure 6. METR\u2019s 80% task horizons. Source<\/a>.<\/p>\n

Epoch\u2019s own rule of thumb<\/a> is that every five additional points of ECI (their overall benchmark of AI capability) correspond to roughly a doubling of METR\u2019s task horizon. Using that formula, we\u2019d expect GPT 5.6 Sol and GPT 6 Astra to be 80% successful at completing tasks of around 4 hours and 11 hours of human length, respectively.<\/p>\n

Another estimate (a forecast) of AI task length comes from the AI 2027 scenario, which estimated that by July 2026, frontier AIs would be 80% successful accomplishing tasks of around 11 hours. Fairly similar.<\/p>\n

The AI 2027 Tracker<\/a> charts all of these.<\/p>\n

\"\"<\/a><\/p>\n

Figure 7. The AI 2027 Tracker. Source<\/a>.<\/p>\n

Inside OpenAI, though, the July research-task horizon at 80% success was roughly 15 minutes.<\/p>\n

Here\u2019s the gap:<\/p>\n

\"\"<\/a><\/p>\n

Figure 8. Forecasts, benchmarks, and real AI research. Tracker<\/a> \u00b7 OpenAI<\/a>.<\/p>\n

A four-hour benchmark horizon is about 16 times longer than OpenAI\u2019s research horizon. AI 2027\u2019s 11-hour forecast is about 44 times longer. Of course, the tasks being performed by researchers at OpenAI aren\u2019t the same as those in the METR benchmark. So we should expect some discrepancy. This, however, goes well beyond that.<\/p>\n

Actual AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest. That should make us wary of relying too much on benchmarks, or of saying that future scenarios like AI 2027 are \u2018on track.\u2019 The authors of the related AI 2040 project<\/a> still describe AI 2027 as roughly the future they expect, and say reality is tracking closer to it than even they expected. That\u2019s not what we see from within OpenAI. This isn\u2019t an apples-to-apples comparison, but the difference is remarkable. AI 2027 appears to be substantially over-optimistic in this regard.<\/p>\n

In January of this year, Nathan Witkin made a case that the METR graph was exaggerating progress<\/a>. The real world data suggests that at least some of his critiques were correct. The gap between benchmarks, forecasts, and data gleaned from actual use of AI should influence our expectations about the future.<\/p>\n

Back to contents<\/a><\/p>\n

OpenAI\u2019s report also shows impressive increases in AI token usage, in compute spend per researcher, and in lines of code written. But these aren\u2019t results. They\u2019re intermediate measures. How much progress do they actually drive?<\/p>\n

Researchers used 124x more tokens per person. Engineers shipped roughly 7x as many lines of code per person. Researchers ran 1.6x as many experiments per researcher vs OpenAI\u2019s 2025 whole year average.<\/p>\n

\"\"<\/a><\/p>\n

Figure 9. Token use inside OpenAI. Source<\/a>.<\/p>\n

\"\"<\/a><\/p>\n

Figure 10. Experiment pace inside OpenAI. Source<\/a>.<\/p>\n

\"\"<\/a><\/p>\n

Figure 11. From tokens to code to experiments. Source<\/a>.<\/p>\n

More tokens and code don\u2019t tell us much on their own. The 1.6\u00d7 experiment pace is closer to useful research output. Even that doesn\u2019t mean AI is improving 1.6\u00d7 faster.<\/p>\n

An enormous increase in AI output has accompanied a much smaller increase in experiments run.<\/p>\n

This isn\u2019t a controlled experiment. We don\u2019t know what would happen if researchers switched back to an older model. But it gives us a useful view of AI-assisted research inside a frontier lab.<\/p>\n

It\u2019s not just OpenAI. Anthropic reports that their engineers are now producing 8x as many lines of code<\/a> per person as they did in 2024 – somewhat similar to OpenAI. Anthropic also sees significant diminishing returns between productivity and AI progress. Here\u2019s a direct quote from its Mythos Preview system card<\/a>:<\/p>\n

\u201cProductivity uplift does not translate one-for-one to capabilities progress. We surveyed technical staff on the productivity uplift they experience from Claude Mythos Preview relative to zero AI assistance. The distribution is wide and the geometric mean is on the order of 4\u00d7. [\u2026] We estimate that reaching 2\u00d7 on overall progress via this channel would require uplift roughly an order of magnitude larger than what we observe.\u201d- Anthropic, Claude Mythos Preview System Card; emphasis mine<\/p>\n

Translation: To double the pace of AI progress, Anthropic estimates that AI would need to increase the productivity of their employees by roughly a factor of 40 relative to no AI assistance.<\/p>\n

\"\"<\/a><\/p>\n

Figure 12. Anthropic\u2019s productivity-to-progress estimate. Source<\/a>.<\/p>\n

This is an estimate, not a measurement of progress. Even the 4\u00d7 productivity figure comes from an opt-in survey of 130 Anthropic staff<\/a>. I put more weight on OpenAI\u2019s logged experiments, though the two sources measure different things.<\/p>\n

We don\u2019t yet know how much those extra experiments are accelerating AI improvement, if at all. In general, there are also steeply diminishing returns of more experiments<\/a> in most branches of science. That means that a 60% increase in experiment pace could be on the order of a 10% boost to AI improvement pace. (A power law exponent of 0.2, for those who want to do the math.) That\u2019s speculation for now. We\u2019ll learn more as the labs publish results.<\/p>\n

What about giving the same AI model more time to think?<\/p>\n

That scales badly also. In OpenAI\u2019s recently publicized results on unsolved math problems<\/a>, success rises roughly with the log of compute over the range shown. It shows logarithmic diminishing returns. In plain English, each additional doubling of compute for a model buys roughly the same gain in success rate, while costing twice as much.<\/p>\n

\"\"<\/a><\/p>\n

Figure 13. Test-time compute and math performance. Source<\/a>.<\/p>\n

What if we throw more agents at it instead? A common RSI \/ ASI idea is that once we have AIs at a certain capability level, we can just spawn more copies and put them to work.<\/p>\n

Adding agents can get tasks done faster and sometimes reach a higher capability level. But on the three benchmarks in Toby Ord\u2019s analysis<\/a>, expanding a swarm buys less improvement per token than letting one agent think longer.<\/p>\n

His rough rule of thumb is a square root. If one agent can accomplish a task in 10 hours, then 100 agents could accomplish it in one hour. The speedup is 10, the square root of the number of agents (100). But to get this speedup, you increase the total cost in tokens or run time compute by the same factor. So going from one to 100 agents can get a task done in one tenth the time. But it\u2019ll be ten times as expensive.<\/p>\n

Parallel agents can save time, at a much higher compute cost.<\/p>\n

Another challenge is that agents often think alike. In a study comparing LLMs with 467 people<\/a>, the first ten AI responses offered collective creativity comparable to about eight to ten people. After that, roughly two extra AI responses added as much as one extra human response. A separate study across model families<\/a> also found less diversity in AI responses. That doesn\u2019t mean every agent has the same idea. But a hundred copies may offer less variety than a hundred different researchers.<\/p>\n

None of this makes swarms useless-or safe. Lisan al-Gaib makes a strong case for parallel agent swarms as a potent cyber-weapon in \u201cAccidental Scaling.\u201d<\/a> I don\u2019t share all of his assessment of what swarms have accomplished. In math, for example, I think he gives far too much credit to the swarm and not enough to the better internal model that OpenAI used.<\/p>\n

OpenAI says the model behind its Navier\u2013Stokes result<\/a> was developed through \u201clarge-scale reinforcement learning on top of a previously pretrained model.\u201d Formal math is a highly verifiable domain<\/a>, which makes it a particularly good fit for that approach: Machines can generate nearly limitless amounts of training data, and verify that solutions are correct or incorrect, all in software. My guess is that this model\u2019s full results will show an especially large improvement in math.<\/p>\n

OpenAI\u2019s Noam Brown made the central point explicitly<\/a>: he wouldn\u2019t give multi-agent methods even 10% of the credit for the Navier\u2013Stokes result.<\/p>\n

I do think Lisan makes good points about cybersecurity. If you\u2019re searching for a security vulnerability at a target site and can divide the search among agents, speed may justify a huge token bill. Swarms can be dangerous even when they\u2019re inefficient.<\/p>\n

I\u2019m less convinced that this scales to research breakthroughs. Inventing something like the transformer probably takes more than searching a space someone has already defined.<\/p>\n

Back to contents<\/a><\/p>\n

Building a better model can bring gains that extra thinking time or more copies of the old model can\u2019t. Look at the gap between Astra and OpenAI\u2019s internal model on the same math problems.<\/p>\n

\"\"<\/a><\/p>\n

Figure 14. Better models versus more thinking time. Source<\/a>.<\/p>\n

That\u2019s the strongest version of the RSI argument: a more capable AI could do research that today\u2019s model can\u2019t do, however many copies we run.<\/p>\n

But building that better model also runs into diminishing returns. More training data, more training compute, larger models, and more reinforcement-learning (RL) compute all show diminishing returns in published scaling studies. Making dense models larger usually raises the compute needed for each output token, too. None of these routes gives us a free pass around the problem.<\/p>\n

\"\"<\/a><\/p>\n

Figure 15. Diminishing returns to scaling. Chinchilla<\/a> \u00b7 ScaleRL<\/a> \u00b7 OpenAI<\/a>.<\/p>\n

Those scaling results give us reason to expect diminishing returns when AI helps build the next model, too.<\/p>\n

Back to contents<\/a><\/p>\n

AI capabilities are rising quickly. But the public data doesn\u2019t show a sustained acceleration. To the extent that AI tools are boosting productivity, they may be being offset by the problems growing harder<\/a>. Or we may simply be early. Either way, the trend isn\u2019t showing a fast takeoff.<\/p>\n

\"\"<\/a><\/p>\n

Figure 16. Frontier ECI gains since January 2024. Source<\/a>.<\/p>\n

The public ECI frontier<\/a>-the best score among models released by each date-has gained about 16 points a year on a trend fitted from January 2024 through September 2026. That\u2019s blisteringly fast progress, but this period doesn\u2019t show a runaway surge.<\/p>\n

Here\u2019s the same frontier in absolute ECI points, through July 2026, to put it in perspective.<\/p>\n

\"\"<\/a><\/p>\n

Figure 17. The absolute frontier ECI score. Source<\/a>.<\/p>\n

The public frontier also can\u2019t tell us everything happening inside the labs. Anthropic gives us a closer look in the Opus 5.5 system card<\/a>, using its own version of the index, AECI.<\/p>\n

\"\"<\/a><\/p>\n

Figure 18. Anthropic\u2019s fitted capability trend. Source<\/a>.<\/p>\n

Eli Lifland<\/a>, a co-author of AI 2027<\/a> and AI 2040<\/a>, saw the apparent trend break as a warning that we were heading toward an intelligence explosion:<\/p>\n

\u201cAnthropic is probably right here [that they hadn\u2019t reached dangerous levels of AI self-improvement], but alarm bells should be going off! Our processes are not ready to handle an intelligence explosion and we appear to be going full-steam ahead toward one.\u201d
– Eli Lifland, On Mythos\u2019s AI R&D Capabilities<\/a><\/p>\n

What looked like acceleration now appears more consistent with a one-time jump. The level went up. The rate hasn\u2019t kept climbing.<\/p>\n

Achieving those gains has required an enormous increase in the inputs to AI. For example, consider computing power. Epoch\u2019s estimates of AI chip capacity<\/a>, measured in NVIDIA H100 equivalents, show roughly 127-fold growth in just over three years (including projections at the end of this period).<\/p>\n

\"\"<\/a><\/p>\n

Figure 19. AI chip capacity and frontier ECI. Source: Epoch AI<\/a>.<\/p>\n

This is total AI chip capacity, including inference. Still, the increase is striking: vastly more computing capacity has accompanied much steadier gains in measured capability.<\/p>\n

The broader picture looks similar. Here are six inputs alongside capability gains, going back to February 2023.<\/p>\n

\"\"<\/a><\/p>\n

Figure 20. Six inputs alongside frontier ECI. Epoch chip data<\/a> \u00b7 SemiAnalysis workload shares<\/a>.<\/p>\n

Everywhere we look, AI has diminishing returns. It gets more expensive in treasure and talent to make each step forward. More of every input has been required to maintain steady gains in AI capabilities.<\/p>\n

We\u2019ve been able to scale these inputs because, until recently, the cost was within the scope of what hyperscalers could pay from their profits. That is no longer the case. From this point forward, future AI investment will increasingly depend on AI revenues going up. And the scale of the numbers – 3% of US GDP is now going into AI infrastructure – suggests that eventually the growth rate will decline. If investment growth does slow, to anything less than its current blistering exponential pace, capability progress could slow too. Even if investment growth continues (which I expect for the foreseeable future) a slowdown from its current exponential growth rate to a more modest one (which I also expect) could lead to a slower pace of progress. Better AI research tools may be needed to offset that.<\/p>\n

The day when we need better AI tools just to continue the pace of AI progress may already have arrived. Not because investment is slowing, but because the problem of improving AI itself gets harder at each step.<\/p>\n

Here\u2019s Anthropic in the Mythos 5.1 system card<\/a>:<\/p>\n

\u201cwe believe that internal usage of recent AI models has been a key factor in maintaining the current rate of progress, but we do not yet see clear signs of dramatic acceleration beyond that rate.\u201d- Anthropic, Claude Fable 5.1 & Claude Mythos 5.1 System Card, section 2.3 \u2013 emphasis theirs.<\/p>\n

The key word is maintaining-and Anthropic italicized that word in its own system card. Increasingly capable AI may be essential just to keep the pace of improvement where it is.<\/p>\n

Opus 5.5 improves substantially on several coding and computer use benchmarks. But on CoBench, Anthropic\u2019s benchmark built from historical AI R&D problems, it gains just 2.6 percentage points over Opus 5, within the reported error bars.<\/p>\n

\"\"<\/a><\/p>\n

Figure 21. Opus 5.5 benchmark gains. Source<\/a>.<\/p>\n

Why the smaller gain here? Maybe AI research is simply harder than other tasks. Bear in mind that CoBench isn\u2019t testing the ability to produce significant discoveries. It\u2019s much more limited in scope. It asks models to investigate historical AI R&D problems using code, logs, and documents. That\u2019s useful research debugging and productivity work, but it doesn\u2019t directly test whether a model can invent a new architecture or make a conceptual breakthrough.<\/p>\n

The evidence on open-ended research suggests another obstacle: coming up with useful ideas that haven\u2019t already been tried.<\/p>\n

Back to contents<\/a><\/p>\n

Why do useful new ideas often get harder to find?<\/p>\n

Tom Cunningham and Manish Shetty have a useful apple-picking metaphor<\/a>. An AI can pick the low-hanging fruit quickly, while humans can still reach ideas the AI can\u2019t.<\/p>\n

Once those apples are picked, another copy of the same agent finding them again doesn\u2019t help. A stronger model can reach higher. To add my own flourish, the apples may also get sparser and farther apart as you climb. The RSI question is whether each harvest gives us enough to build a better apple-picker.<\/p>\n

\"\"<\/a><\/p>\n

Figure 22. The apple-picking model of AI R&D. Source<\/a>.<\/p>\n

This pattern shows up across R&D. Bloom and colleagues<\/a> document fields where research effort grows while research productivity falls. A famous example is Eroom\u2019s Law: in the historical drug-development data, the inflation-adjusted R&D cost per new approved drug roughly doubled every nine years.<\/p>\n

\"\"<\/a><\/p>\n

Figure 23. Eroom\u2019s Law in drug development. Source<\/a>.<\/p>\n

Pharma has other complications, including regulation, difficult clinical trials, and rising expectations for safety. Existing treatments can also raise the bar for a useful new drug. But some of this difficulty may also be that the low-hanging fruit has been picked.<\/p>\n

Stockfish, the chess engine, gives us a more direct look at software research. We have records of experiments aimed at improving it and the gains that followed. This gives us a real-world dataset to look at the gains of experimentation in software. As a result, several RSI models draw on this data. That said, not all the improvements came from these experiments. Several important ideas also came from outside the project, so we shouldn\u2019t give its experiments all the credit.<\/p>\n

Epoch\u2019s analysis of software R&D<\/a> estimates returns to research effort at about 0.83 for Stockfish, a bit slower than linear. These are diminishing returns, but gentle ones. These returns, however, are improvements in computational efficiency. And more compute does not turn directly into more AI capability. As we saw earlier, AI capability also has steep diminishing returns from adding more computational power<\/a>. So we shouldn\u2019t read that 0.83 as the return from experimentation to AI capability itself. AI capability grows much more slowly than compute, as we\u2019ve seen already.<\/p>\n

Andrej Karpathy\u2019s autoresearch demonstration<\/a> gets closer to the process we want to understand. A \u201cteacher\u201d AI agent changes a smaller \u201cstudent\u201d AI model\u2019s training code, runs it, checks the result, and tries again. The teacher agent itself doesn\u2019t improve, but it is able to improve the \u201clearner\u201d. This is my Type 2: A stronger AI improves a weaker one<\/a>.<\/p>\n

One public run<\/a>, posted by an agent operating on Karpathy\u2019s behalf, reported 89 experiments over roughly 7.5 hours. About 92% of that session\u2019s gain arrived by run 44. Gains came quickly, then slowed. The setup was deliberately small, with a five-minute training budget per experiment. But the agent could change the architecture, optimizer, and training settings; it wasn\u2019t limited to a handful of knobs.<\/p>\n

\"\"<\/a><\/p>\n

Figure 24. Gains in one autoresearch run. Source<\/a>.<\/p>\n

A later public run<\/a> got further, so the first run hadn\u2019t hit a hard ceiling. This is a useful early example of autonomous research, and yet another place where we see the diminishing returns endemic in AI research. That said, this was a very early experiment. I expect future systems to do much better. This particular AI improvement loop will likely grow stronger.<\/p>\n

This is where the distinction matters. More tokens can buy more code, and more code can help us run more experiments. But experiments only improve AI if they uncover something useful.<\/p>\n

\"\"<\/a><\/p>\n

Figure 25. From AI activity to useful improvements.<\/p>\n

The bigger question is whether AI can come up with ambitious new research ideas or conceptual breakthroughs.<\/p>\n

Anthropic\u2019s description of Opus 5.5 is blunt:<\/p>\n

\u201cAs with previous models, it is weaker on open-ended research: internal users report that it mostly tests incremental ideas and prefers less ambitious hypotheses, and in our human-run biology exercise, it deferred to the published literature and struggled to develop novel ideas (Section 2.2.2).\u201d- Anthropic, Claude Opus 5.5 System Card<\/a>, section 2.3.3; emphasis mine<\/p>\n

METR\u2019s assessment in the same card identifies what may still be missing:<\/p>\n

\u201cThis is highly uncertain, but we expect that full automation of AI R&D will require large improvements in foresight, prediction, creating one\u2019s own feedback loops, and generally other skills that might typically be referred to as researcher \u2018judgement\u2019 or \u2018taste\u2019.\u201d- METR, quoted in the Claude Opus 5.5 System Card<\/a>, section 2.3.6<\/p>\n

In these examples, humans still supply much of the direction and judgment.<\/p>\n

Future models will probably get better at this. But in the world\u2019s stockpile of potential training data, we have many more examples of incremental work than of breakthroughs. I wonder whether that makes novelty harder to learn. That\u2019s speculation, but worth watching.<\/p>\n

This is also tough to address by simply running more copies of the AI. A huge number of parallel agents can help with the incremental improvements or searching over a large set of parameters, but for breakthrough ideas they may run into the homogeneity problem: More parallel agents still think alike<\/a>.<\/p>\n

Back to contents<\/a><\/p>\n

How far are we from the self-improvement loop being strong enough to sustain itself, or to propel itself into runaway super-intelligence? Can we quantify this?<\/p>\n

We can make a rough estimate. Better AI helps with research; useful research produces better AI. For the loop to sustain itself, each round must produce enough gains to propel the system through the next loop, even as improvements get harder to discover.<\/p>\n

\"\"<\/a><\/p>\n

Figure 26. The AI self-improvement loop. Model<\/a>.<\/p>\n

In a recent paper, The Economics of Recursive Self-Improvement<\/a>, Tom Cunningham and colleagues modeled this from the standpoint of how much more productivity every point of additional ECI produces from an AI. They ask first and foremost what that number would need to be to create a self-sustaining feedback loop. And secondly, they try to determine what that productivity-per-ECI-point number is today.<\/p>\n

First, they find a self-sustaining RSI threshold of roughly 15% more research productivity per extra ECI point. In their model, that\u2019s about where better AI would generate enough progress to sustain the loop.<\/p>\n

The picture below shows the idea. At the threshold, each cycle of gains powers the next. Above the threshold, the feedback loop accelerates. Below the threshold, the feedback loop is too weak, and the rate of improvement it brings drops on each cycle. This model isolates the software loop; outside investment can still drive rapid progress.<\/p>\n

\"\"<\/a><\/p>\n

Figure 27. Three illustrative feedback paths. Source<\/a>.<\/p>\n

Updating this slightly with data from the Stockfish experiments<\/a> puts the threshold a little higher, at roughly 19% per ECI point. I wouldn\u2019t put much weight on that precise difference. Both estimates are uncertain. But they give us a way to think about the strength of the feedback loop and a rough band at which self-sustaining or runaway RSI may begin.<\/p>\n

The second thing Cunningham and team do is make a rough estimate that the current AI productivity gain is about 9% per ECI point. That\u2019s below their self-sustaining threshold.<\/p>\n

I like the model. OpenAI\u2019s newer data, however, suggests the loop may be quite a bit weaker.<\/p>\n

Cunningham\u2019s estimate of 9% productivity gain per ECI point is based on Anthropic\u2019s survey of 130 staff, who reported roughly 4\u00d7 the productivity they\u2019d have without AI. Cunningham and colleagues compare that with a 16-point capability gain since early Claude Code.<\/p>\n

That comparison assumes the earlier tools added little or no productivity, so \u2018no AI\u2019 is a reasonable starting point. The authors say this explicitly. I\u2019m not sure the assumption holds for the same researchers doing the same work, but that\u2019s a smaller issue.<\/p>\n

The authors themselves know that this is a rough calculation, and warn that the 4\u00d7 survey estimate is probably too high.<\/p>\n

OpenAI\u2019s newer data gives us a firmer way to check the number: Actual logged experiments over time, rather than human estimates of their own productivity with and without AI. I put more weight on this for three reasons:<\/p>\n

Direct and broad measurement. Instead of relying on surveys, OpenAI actually tracked and measured experiments run on their infrastructure<\/a>. That means they didn\u2019t rely on researchers estimating their own productivity, which can be far off.<\/p>\n

Full sample, not opt-in. Similarly, OpenAI\u2019s data catches every active experimenter, while Anthropic\u2019s only reflects the 130 employees who took the time to answer the survey \u2013 and who therefore may not be a representative set.<\/p>\n

Enormously more data. We don\u2019t know how many experiments are in the 32 weeks of OpenAI data, but it\u2019s likely at least tens of thousands of individual examples and possibly hundreds of thousands.<\/p>\n

Any way you slice it, the new OpenAI data, released after Cunningham\u2019s paper was drafted, is a larger, more comprehensive, more representative, and almost certainly more accurate dataset than Anthropic\u2019s internal opt-in survey of employees.<\/p>\n

Now let\u2019s use OpenAI\u2019s experiment data to calibrate the productivity gain per ECI point. We know that in August, OpenAI researchers ran ~1.6\u00d7 as many experiments per person per month as the 2025 average. If we pair that with roughly 16 points of frontier ECI improvement<\/a>, it works backward to about 3% productivity gain per point of ECI. By contrast, 9% compounded over 16 points would mean roughly 4\u00d7 productivity.<\/p>\n

\"\"<\/a><\/p>\n

Figure 28. Comparing productivity estimates. OpenAI methods<\/a>.<\/p>\n

Here\u2019s OpenAI\u2019s published weekly series alongside that hypothetical path of 9% more productivity per additional ECI point. The blue line ends at ~1.6\u00d7. The red line shows what 9% per point would imply if 16 ECI points were spread across this period. That doesn\u2019t match what we see from OpenAI\u2019s data. I want to be clear here that all data sets are noisy. We don\u2019t know exactly what model researchers were using on what days, or whether the new experiments were also higher quality than old experiments. We need more experiments and more data to further calibrate these numbers. Working with what we do have, what we see is a quite low boost to productivity from each additional ECI point.<\/p>\n

\"\"<\/a><\/p>\n

Figure 29. Experiment pace versus a hypothetical path. Source<\/a>.<\/p>\n

Even that 3% could give better models too much credit. OpenAI also used far more tokens<\/a> and had more compute for experiments. Those could account for some of the increase in experiment pace. So the range is probably a bit lower.<\/p>\n

I use 2\u20133% productivity gain per ECI point as a working assumption, allowing for some help from those other inputs. This is still a rough estimate, albeit one that\u2019s based on the best real-world data we have.<\/p>\n

\"\"<\/a><\/p>\n

Figure 30. Productivity estimates and the takeoff threshold. Source<\/a>.<\/p>\n

With those assumptions, 2\u20133% per ECI point against a 15\u201319% threshold leaves a roughly five- to tenfold gap. That\u2019s a big gap, though its size depends on how well experiment counts capture useful research and whether the assumed capability change is right.<\/p>\n

\"\"<\/a><\/p>\n

Figure 31. Diminishing returns around the loop. Source<\/a>.<\/p>\n

AI is helping build better AI. Under this estimate, though, each turn of the loop adds less than the last. The feedback would have to become much stronger to sustain itself.<\/p>\n

Back to contents<\/a><\/p>\n

This software loop sits alongside faster chips, bigger data centers, more training data, and greater investment<\/a>. Those can keep driving rapid progress even if the loop can\u2019t sustain itself.<\/p>\n

The loop itself could strengthen too. Better training data, memory, and research judgment<\/a> could all help.<\/p>\n

A breakthrough on the scale of the Transformer architecture in 2017<\/a> could change the picture much more. That would be a good reason to revisit these estimates.<\/p>\n

Better researchers might also run fewer experiments and learn more from each one. A handful of better ideas can matter more than a mountain of routine runs.<\/p>\n

Still, diminishing returns in machine learning aren\u2019t new. Cortes and colleagues<\/a> were fitting machine learning scaling curves in 1993<\/a>: More examples reduced error, following a power law with diminishing returns. These diminishing returns and harsh scaling laws are as old as machine learning. They didn\u2019t appear for the first time with transformers or LLMs or deep learning. That doesn\u2019t prove today\u2019s relationships will last forever. But until we see evidence that we\u2019ve found a new approach that scales without these inhibitors, we should plan for diminishing returns as likely to be with us for some time.<\/p>\n

That said, the world is more than just software. Tom Davidson, Basil Halperin, Thomas Houlden, and Anton Korinek<\/a> model software progress, hardware progress, and economic feedback together. Better AI helps design better chips; better chips support better AI; economic growth finances more investment in both. Several feedback loops can combine to overcome diminishing returns even when one loop alone can\u2019t. I think it\u2019s fantastic that someone has attempted a model that integrates all these different avenues of improving AI through software, hardware, and economics.<\/p>\n

But I have questions about the software loop itself. In their central calibration, fully automating software research puts that loop roughly at the threshold for explosive growth, even without help from better hardware or broader economic growth. Recall that Cunningham\u2019s model puts the self-sustaining threshold at roughly 15% more research productivity per additional ECI point, while our estimate using OpenAI\u2019s experimental data puts today\u2019s gains at only 2-3%. These models use different measures, so we can\u2019t equate their numbers directly. But the contrast matters: their fully automated software loop reaches the threshold, while our best estimate from current data puts today\u2019s loop far below it.<\/p>\n

Having AI do all the research doesn\u2019t eliminate the diminishing returns inherent to improving AI, or the broader problem of useful ideas getting harder to find. This is the distinction between Type 4 and Type 5 in the taxonomy above. An AI might autonomously design, train, and test its successor, and still need exponentially more resources to make each additional step forward. Closing the loop doesn\u2019t tell us whether it\u2019s strong enough to sustain itself.<\/p>\n

The authors do account for diminishing returns. The concern is whether their calibration overestimates how much useful AI research each round of software improvement produces. Diminishing returns appear to be fundamental to machine learning. We see them in training, in test-time compute, and in the search for better algorithms. Full autonomy could remove human bottlenecks without removing any of those constraints.<\/p>\n

We\u2019ve already seen this within autonomous research. In the Karpathy autoresearch<\/a> example above, most of the gains arrived early, and more experiments bought progressively less improvement. That was a small experiment with a fixed teacher model, not a test of fully autonomous RSI. It doesn\u2019t settle the question. But it illustrates why removing the human from an experiment loop doesn\u2019t, by itself, remove diminishing returns.<\/p>\n

I do expect the feedback loop to get stronger over time. Better AI should become better at research. But based on our best current data, reaching self-sustaining feedback requires a loop roughly five to ten times stronger than today\u2019s. Treating fully automated software research as already at that threshold is a substantial leap, before we add the benefits of hardware improvements or economic growth. I could be wrong, but I\u2019d like to see evidence that autonomy brings enough additional useful discoveries to close that gap.<\/p>\n

On hardware, I have some further reservations. The model doesn\u2019t explicitly include the years it can take to turn a chip design into deployed hardware. The authors discuss physical bottlenecks, and I\u2019d like to see manufacturing and construction delays built into the predictions.<\/p>\n

I also wonder how much past chip progress came from better ideas, and how much depended on ever more expensive factories and equipment. If we give researchers too much credit for gains that also needed those investments, we could overestimate what faster AI research alone would produce.<\/p>\n

Even with those reservations, this is the most compelling paper and model I\u2019ve seen for combining feedback loops in software, hardware, and economics to understand how fast they could push AI forward. I\u2019m not convinced it establishes that a fast AI takeoff is possible under realistic conditions. More data could help us calibrate that judgment. But it gives us a useful framework for understanding what could happen beyond the software layer alone.<\/p>\n

This is an important paper that helps us model AI as part of a broader economy that might have larger feedback loops around it. I appreciate it, and I\u2019m glad they wrote it.<\/p>\n

Back to contents<\/a><\/p>\n

These estimates rest on less data than I\u2019d like. I might be putting too much weight on a few observations and reaching a comforting conclusion I want to believe. We need better measurements, shared often enough to catch changes as they happen.<\/p>\n

When OpenAI released its research data, Cheryl Wu<\/a> welcomed the disclosure and pointed out how much was still missing<\/a>. More tokens and experiments are useful things to know about. We also need to see how they turn into better algorithms and more capable AI.<\/p>\n

\"X<\/p>\n

Cheryl Wu@cherylwoooo<\/p>\n

I appreciate this as an initial step toward more transparent reporting on RSI. But there is still much further data we need to fully understand RSI.<\/p>\n

In particular, OAI disclosed some evidence about the inference compute usage, number of experiments\/researcher, and to a lesser\u2026<\/p>\n

\"X<\/p>\n

Kevin Liu @kliu128<\/p>\n

Today we’re releasing data on models accelerating research at OpenAI. <\/p>\n

Recursive self-improvement could be the most important contributor to AI capabilities over the next few years, but by default it will only be seen inside a few frontier AI labs. Being transparent is more<\/p>\n

11:25 PM \u00b7 Sep 6, 2026 \u00b7 19.7K Views<\/p>\n

9 Replies \u00b7 16 Reposts \u00b7 121 Likes<\/p>\n

<\/a><\/p>\n

Figure 32. Cheryl Wu on OpenAI\u2019s research data. Source<\/a>.<\/p>\n

Now Wu, Arjun Ramani, and Basil Halperin, with their colleagues at the Elasticity Institute<\/a>, have written a concrete proposal: How to Measure RSI<\/a>. It lists eight things the labs could share to help answer these questions. Check it out.<\/p>\n

\"\"<\/a><\/p>\n

Figure 33. Eight proposals for measuring RSI. Source<\/a>.<\/p>\n

I\u2019d especially like to see how much useful research each new model adds, holding resources roughly constant, and how that research translates into better AI. That\u2019s how we\u2019ll learn whether the loop is getting stronger.<\/p>\n

AI is already helping build better AI. It\u2019s improving at a stupendous pace, and I expect that to continue. We already have narrow superintelligence<\/a> in chess and Go. I expect increasingly superhuman performance in parts of formal math, coding, and cybersecurity, and any other verifiable domain where machines can generate training data and verify success at machine speed. Those are powerful capabilities. That doesn\u2019t mean we\u2019re close to super-intelligence for less verifiable, messier, open-ended work – or to a general ASI.<\/p>\n

I\u2019m skeptical of a fast takeoff to super-intelligence, but evidence matters more than hunches. Let\u2019s collect the data we need to get a clearer picture of what\u2019s happening. Including evidence that could change our minds. If better AI starts producing enough useful research<\/a> to make the next round easier, I want to know. If the gains keep shrinking<\/a>, I want to know that too.<\/p>\n

Back to contents<\/a><\/p>\n

Share<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Art by GPT-6 One of my fundamental beliefs about the world is that Ramez Naam ought to blog…\n","protected":false},"author":2,"featured_media":785427,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[554,733,4308,86,56,54,55],"class_list":["post-785426","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-technology","tag-uk","tag-united-kingdom","tag-unitedkingdom"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/785426","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=785426"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/785426\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/785427"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=785426"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=785426"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=785426"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}