are large language models making us less productive?

This article is an on-site version of The AI Shift newsletter. Premium subscribers can sign up here to get the newsletter delivered every Thursday. Standard subscribers can upgrade to Premium here, or explore all FT newsletters

Welcome back to The AI Shift, our weekly newsletter about AI and the labour market.

If the threat of an entry level job-pocalypse (see last week’s edition) is many people’s biggest worry about generative AI, the other side of that coin should be huge productivity gains for those of us fortunate enough to already have white-collar jobs. So, are experienced workers in the most AI-exposed industries accomplishing more in less time by using their very own intern? Let’s start with a look at the data.

John writes . . .

The first question we need to ask when dealing with AI and productivity is, do the touted gains originate in self-reported guesstimates, or are they directly measured?

A lot of the splashiest headlines fall into the former category. The latest example came only two days ago when we were sent a press release declaring that new research shows “AI boosts productivity by the equivalent of one workday per week”.

A 20 per cent jump in productivity would show up clear as day in the macroeconomic data, but we see nothing of the sort. The devil is in the detail: the figure was arrived at by asking thousands of employees how many hours per week they believe their use of AI saves them in their job. This is certainly not useless information but nor is it especially robust. In fact that 20 per cent figure rang a bell for me . . .

A few months ago, AI research organisation METR ran an experiment to measure AI’s impact on productivity. It recruited 16 experienced software developers and asked them to provide examples of coding tasks on their to-do lists then randomly assigned them to complete some of these tasks using the latest AI tools and others without. The participating developers were then asked how much time they thought AI had saved them. The answer? That familiar 20 per cent. The catch? As well as self-reported estimates, METR recorded how long the tasks had actually taken. It turned out the developers were 19 per cent slower with AI than without.

Some content could not load. Check your internet connection or browser settings.

Based on follow-up conversations with the participants, one plausible explanation METR landed on was that it was precisely the developers’ wealth of expertise that produced the apparent AI slowdown. They were typically working on very large and long-running projects which reward tacit knowledge over AI’s approach of starting from first principles. Their highly trained eyes made them especially fastidious when checking AI-generated code. This complements other research which finds that AI often does boost productivity with more basic code writing and checking. Experience, complexity and context matter.

But METR’s is not the only work casting doubt on some of the most bold claims about AI-enhanced productivity in software. If the modified theory is that we should see more of an impact on lower-complexity code and new projects rather than mature ones, we might expect to see an uptick in the release of new smartphone apps, websites or open-source coding projects. Yet a comprehensive analysis by veteran software engineer Mike Judge found no such thing.

It’s worth re-stating that METR’s was a relatively small and idiosyncratic study which made no claims of being representative of software work as a whole let alone the broader economy, and Judge’s is just another of many ways of looking at this, but two things I think we learn here are 1) people are often not great judges of whether AI is making them more productive, let alone by how much and 2) it’s possible that a tool can feel slick and enjoyable to use without actually making us more efficient.

Sarah writes…

It’s good to be reminded, John, that when it comes to our own work performance, we aren’t always reliable witnesses! But while you’ve looked into this question at the level of individual tasks, I think the story gets even more interesting when you consider productivity at the organisational level. Because even if large language models do help individuals to race through certain tasks, that doesn’t mean businesses should expect an automatic and commensurate increase in overall productivity. That’s because organisations aren’t just groups of individuals, they’re systems.

Let’s stick with coding for a moment because it’s a good example. I found the latest annual report from Google Cloud’s DORA research programme (which surveys software engineers and tracks performance metrics) interesting on this theme. It argues that AI’s role in software development is proving to be that of an amplifier, which “magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones.” The research found that the best returns were generated by teams with a good “underlying organizational system” whereas “without this foundation, AI creates localized pockets of productivity that are often lost to downstream chaos.” As W. Edwards Deming, the legendary management expert once said, “a bad system will beat a good person every time.”

Jason Gorman, who trains and mentors software developers, explained it to me this way: speeding up code generation “in a process with real bottlenecks” can make those bottlenecks worse. “The high-performing teams have addressed the real bottlenecks — so they don’t do things in big batches and phases, with handovers, testing, code review, integration and customer feedback happening continuously,” he said. “Most dev teams need to fix those problems before they even think about attaching a code-generating fire hose to their plumbing.”

I’d be curious to know if this resonates with readers in other professions (do email us). You might be able to draft a memo about a new project faster, but do you still have to wait for a committee to read and approve it? And are those decision-makers now drowning in more memos than ever? There’s a link here, too, with the term “workslop”, coined in a Harvard Business Review article to describe “low-effort, passable looking work that ends up creating more work for their coworkers”.

The example of software development is encouraging overall, though. It suggests that productivity gains should increase over time, as the tools themselves get better, workers learn how to use them more effectively and organisations begin to restructure some processes to make the most of them. The 2025 DORA report suggests that progress has been made since 2024, for example. And if it’s right to say that pre-existing organisational structures can hold things back, then we should start to see some very productive start-ups which can create AI-integrated workflows from a blank slate.

So what have we learned?John

Always check the label on claimed productivity boosts. Self-reported estimates of time saved by AI are an unreliable metric that at best tend to produce over-optimistic exaggerations and at worst turn a negative into a positive. Direct measures of changes in output and time spent are the gold standard, though as both the METR example and your reporting make clear, Sarah, even here we should not assume that results for a particular setting or firm will necessarily apply more broadly. Much as with last week’s topic of job displacement, there is often an assumption that we will quickly see the impact of AI in the form of a broad wave visible in the macroeconomic data, but even if the technology improves smoothly and steadily, its implementation will not.

Sarah

For businesses wondering why they aren’t seeing big productivity gains from AI yet, the best idea is probably to make a careful assessment of systems, workflows and bottlenecks, rather than chivvying individual staff to use the new tools more often. That said, not all bottlenecks are easy to dispatch with. In high-stakes professions like accountancy and law, the requirements for quality and accuracy strike me as one example. John, you point out that AI is becoming quite adept at performing “code review” in the tech sector, but in non-code contexts, human experts are still having to check AI-written documents for hallucinations and errors. We’ve already seen some examples of the reputational and financial consequences for companies which failed to invest sufficient time and expertise in verifying AI-generated output.

There’s still time to save your spot at the FT’s Future of AI Summit on November 5-6, featuring Nvidia’s Jensen Huang and 600 business leaders involved in AI application. As an AI Shift subscriber, you can get 20 per cent off an in-person pass at this year’s summit using the code FT20. Or register for your complimentary digital pass here.

Tags: