How do you translate ancient Palmyrene script from a Roman tombstone? How many paired tendons are supported by a specific sesamoid bone in a hummingbird? Can you identify closed syllables in Biblical Hebrew based on the latest scholarship on Tiberian pronunciation traditions?<\/p>\n
These are some of the questions in \u201cHumanity\u2019s Last Exam\u201d, a new benchmark introduced in a study<\/a> published this week in Nature. The collection of 2,500 questions is specifically designed to probe the outer limits of what today\u2019s artificial intelligence (AI) systems cannot do.<\/p>\n The benchmark represents a global collaboration of nearly 1,000 international experts across a range of academic fields. These academics and researchers contributed questions at the frontier of human knowledge. The problems required graduate-level expertise in mathematics, physics, chemistry, biology, computer science and the humanities. Importantly, every question was tested against leading AI models before inclusion. If an AI could not answer it correctly at the time the test was designed, the question was rejected.<\/p>\n This process explains why the initial results looked so different from other benchmarks. While AI chatbots score above 90% on popular tests<\/a>, when Humanity\u2019s Last Exam was first released in early 2025, leading models struggled badly. GPT-4o managed just 2.7% accuracy. Claude 3.5 Sonnet scored 4.1%. Even OpenAI\u2019s most powerful model, o1, achieved only 8%.<\/p>\n The low scores were the point. The benchmark was constructed to measure what remained beyond AI\u2019s grasp. And while some commentators have suggested<\/a> that benchmarks like Humanity\u2019s Last Exam chart a path toward artificial general intelligence, or even superintelligence \u2013 that is, AI systems capable of performing any task at human or superhuman levels \u2013 we believe this is wrong for three reasons. <\/p>\n Benchmarks measure task performance, not intelligence<\/p>\n When a student scores well on the bar exam, we can reasonably predict they\u2019ll make a competent lawyer. That\u2019s because the test was designed to assess whether humans have acquired the knowledge and reasoning skills needed for legal practice \u2013 and for humans, that works. The understanding required to pass genuinely transfers to the job.<\/p>\n But AI systems are not humans preparing for careers. <\/p>\n When a large language model scores well on the bar exam, it tells us the model can produce correct-looking answers to legal questions. It doesn\u2019t tell us the model understands law, can counsel a nervous client, or exercise professional judgment in ambiguous situations. <\/p>\n The test measures something real for humans; for AI it measures only performance on the test itself.<\/p>\n Using human ability tests to benchmark AI is common practice, but it\u2019s fundamentally misleading. Assuming a high test score means the machine has become more human-like is a category error, much like concluding that a calculator \u201cunderstands\u201d mathematics because it can solve equations faster than any person.<\/p>\n Human and machine intelligence are fundamentally different<\/p>\n Humans learn continuously from experience. We have intentions, needs and goals. We live lives, inhabit bodies and experience the world directly. Our intelligence evolved to serve our survival as organisms and our success as social creatures. <\/p>\n But AI systems are very different<\/a>.<\/p>\n Large language models derive their capabilities from patterns in text during training. But they don\u2019t really learn<\/a>. <\/p>\n For humans, intelligence comes first and language serves as a tool for communication \u2013 intelligence is prelinguistic<\/a>. But for large language models, language is the intelligence \u2013 there\u2019s nothing underneath.<\/p>\n Even the creators of Humanity\u2019s Last Exam acknowledge<\/a> this limitation:<\/p>\n High accuracy on [Humanity\u2019s Last Exam] would demonstrate expert-level performance on closed-ended, verifiable questions and cutting-edge scientific knowledge, but it would not alone suggest autonomous research capabilities or artificial general intelligence.<\/p>\n Subbarao Kambhampati, professor at Arizona State University and former president of the Association for the Advancement of Artificial Intelligence, puts it more clearly<\/a>: <\/p>\n Humanity\u2019s essence isn\u2019t captured by a static test but rather by our ability to evolve and tackle previously unimaginable questions.<\/p>\n Developers like leaderboards<\/p>\n There\u2019s another problem. AI developers use benchmarks to optimise their models for leaderboard performance. They\u2019re essentially cramming for the exam. And unlike humans, for whom the learning for the test builds understanding, AI optimisation just means getting better at the specific test.<\/p>\n But it\u2019s working. <\/p>\n Since Humanity\u2019s Last Exam was published online in early 2025, scores have climbed dramatically<\/a>. Gemini 3 Pro Preview now tops the leaderboard at 38.3% accuracy, followed by GPT-5 at 25.3% and Grok 4 at 24.5%.<\/p>\n Does this improvement mean these models are approaching human intelligence? No. It means they\u2019ve gotten better at the kinds of questions the exam contains. The benchmark has become a target to optimise against.<\/p>\n The industry is recognising this problem. <\/p>\n