Benchmarks are the rulers of AI, in both senses of the word. As measuring sticks, they work according to an extremely simple logic: If you want to know what an AI model can do, devise a standardized test to gauge it. But AI benchmarks can also dominate the behavior of the entire field.<\/p>\n
In 1987, the Defense Advanced Research Projects Agency\u2019s first benchmark<\/a> on automatic speech-to-text translation showed that state-of-the-art AI could not outperform so-called \u201cdumb\u201d statistical approaches. The poor performance helped to usher in an AI winter. For 25 years, interest in AI research waned and funding dried up.<\/p>\n Then, in 2012, another benchmark<\/a> called ImageNet swung the pendulum back. An eight-layer neural network trounced<\/a> other algorithms at classifying images and kicked off a \u201cdeep learning revolution.\u201d<\/p>\n Nowadays, new AI benchmarks, geared at large language models such as OpenAI\u2019s GPT-6 Astra, get created and solved with clockwork regularity, purporting to measure everything from domain-specific expertise<\/a> to \u201cgeneral intelligence<\/a>.\u201d But rather than hail this trend as proof of AI\u2019s skyrocketing potential, many in the field \u2014 skeptics<\/a> and boosters<\/a> alike \u2014 take it as evidence that benchmarking itself has become fatally flawed<\/a>.<\/p>\n