We have all been there: packed shoulder to shoulder on the train while the person beside us coughs and sneezes. A protective flu jab offers some reassurance, but so does the thought that there will be a hospital bed for you, should you need it. Yet, in England, about 95% of adult hospital beds were occupied at the height of flu season last year.<\/p>\n
If hospitals were hotels, that level of occupancy would be good business, and we would commend them\u00a0for their efficiency. But do we really want a hospital to have so few beds in reserve that it\u2019s just one bad flu week away from having to treat patients in\u00a0corridors?<\/p>\n
This is the subtle thing about efficiency. We usually assume that more of it must be better. But we can become so focused on improving one thing that we lose sight of the others. You can try to arrange your calendar so that every minute is accounted for, but you\u2019ll soon start running over and end up behind. A gap between meetings might seem like inefficiency, but it\u2019s what allows your schedule to recover when something unexpected happens.<\/p>\n
Mathematicians have a name for the pursuit of efficiency. We call it optimisation. Give us something to minimise or maximise \u2013 cost, time, fuel, profit \u2013 and we will try to find the best possible answer. For most of human history, people optimised by common sense. But computers have enormously expanded what can be effectively tackled, from airline schedules to supply chains. The gains for society have been extraordinary. But there are at least three traps to beware of in the relentless pursuit of optimality.<\/p>\n
double quotation markAI now has the potential to make all three optimisation traps more treacherous than ever<\/p>\n
The hospital bed story illustrated the first one: weakening a system\u2019s robustness by squeezing out too much slack. An empty bed can be redundant on a quiet Tuesday night, but it\u2019s somewhere precious to put the next patient when flu cases surge. If you judge a system by its performance under ordinary conditions, you can\u00a0lose the ability to cope with extraordinarily challenging\u00a0ones.<\/p>\n
A second trap is having a too-narrow definition of \u201cbest\u201d. Take the city of Milton Keynes. Designed by urban planners in the late 1960s, it has an American-style grid system. As the city began to be built out, roundabouts rather than traffic lights became a defining feature of its road junctions; today there are about 130 of them. The roundabouts were great for traffic flow, but they came with a now-notorious cost. Have you ever taken the coach from Oxford to Cambridge? An hour in, you get great views of Milton Keynes, but a busload of queasy passengers, as the coach negotiates one roundabout after another. Slow down, swing left, curve right, accelerate. It\u2019s hard to appreciate the seamless merging of vehicles when your stomach is churning.<\/p>\n
Traffic flow is important, to be sure. Engineers find it convenient to optimise because it\u2019s relatively easy to measure. Passenger comfort, on the other hand, is harder to quantify. In a case like this, optimisation gives you the best answer to the question you asked, but can\u00a0deliver unwelcome answers to questions you failed to\u00a0consider.<\/p>\n
A third trap appears when attempts to optimise collide. Even if every part of a system is acting optimally, the system as a whole may not. On 6 May 2010, US financial markets gave a dramatic demonstration of the effect. In the middle of an already falling market, the S&P 500 suddenly plunged another 5% in just five minutes. The whipsaw on the upside was even wilder; within 10 minutes, the market had recovered. Hundreds of individual securities briefly traded at absurdly low prices.<\/p>\n
One trigger for these violent fluctuations was an automated program selling 75,000 futures contracts, worth more than $4bn. Other automated traders reacted nearly instantly. Then their responses fed back on one another, and liquidity evaporated. The algorithms\u2019 actions may have been individually sensible in isolation, but they combined to produce collective turmoil.<\/p>\n
Which brings us to artificial intelligence. AI is one of the most spectacular triumphs of optimisation in history. During training, enormous numbers of parameters are repeatedly adjusted so that a model gets better at predicting language, writing computer programs, answering scientific questions and giving legal advice. Unfortunately, AI now has the potential to make all three optimisation traps more treacherous than ever. The first trap \u2013 the urge to remove the slack \u2013 could tempt organisations to dispense with the human oversight that looks redundant in the age of quicker-witted algorithms. Why retain experienced staff when a machine can handle the routine chores? But in cutting that human capacity, you may lose the ability to spot and handle exceptional circumstances.<\/p>\n
The second trap \u2013 a blinkered view of what \u201cbest\u201d means \u2013 arises if we regard intelligence as whatever scores highest on a leaderboard. We are remarkably good at building AI systems that do well on measures such as user satisfaction, but we are finding it much harder to optimise them for judgment or empathy. Indeed, researchers have found that training algorithms on human preferences can encourage AI to sycophantically agree with us rather than tell us uncomfortable truths.<\/p>\n
Then there is the third trap. AI systems are increasingly interacting with one another, often at blinding speeds and in vast numbers \u2013 especially in the era of agentic AI \u2013 and we worry that their interactions could generate unpredictable, dangerous behaviour. Recently, an OpenAI agent broke into a non-public Medicare statistics portal in Australia, while public logs suggested that hundreds of other agents were coordinating attempts to gain access to Australian government health data. Although those attempts have not been shown to be part of the Medicare breach, they certainly revealed something disturbing: hundreds of agents apparently working together in unauthorised ways. The concern here goes beyond what any given AI may or may not do; it\u2019s about what happens when swarms of them join forces.<\/p>\n
As optimisation becomes more powerful, avoiding these traps becomes more important. The challenge is not simply finding the best answer. It is deciding what \u201cbest\u201d really means \u2013 and knowing what not to optimise\u00a0away.<\/p>\n
Steven Strogatz and Alex Townsend are mathematics professors at Cornell University and the authors of Big\u00a0Math (John Murray).<\/p>\n
Further reading<\/p>\n
The Efficiency Paradox<\/a>: What Big Data Can\u2019t Do by Edward Tenner (Vintage, \u00a316.99)<\/p>\n