{"id":435380,"date":"2026-05-10T22:56:15","date_gmt":"2026-05-10T22:56:15","guid":{"rendered":"https:\/\/www.newsbeep.com\/il\/435380\/"},"modified":"2026-05-10T22:56:15","modified_gmt":"2026-05-10T22:56:15","slug":"misplaced-panic-over-ai-progress","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/il\/435380\/","title":{"rendered":"Misplaced panic over AI progress"},"content":{"rendered":"

\"\"<\/a>1. Take the mostly horizontal part (yellow dots, actually attested); 2. Infer vertical.<\/p>\n

A couple days ago METR, a think tank that evaluates AI, dropped its latest graph, and the Twitterverse quickly became overwhelmed with panic, including a pile of tweets like these (and the one above):<\/p>\n

\"\"<\/a><\/p>\n

and <\/p>\n

\"\"<\/a><\/p>\n

All were triggered by METR\u2019s latest edition of their famous \u201ctime horizon\u201d graph:<\/p>\n

\"X<\/p>\n

METR@METR_Evals<\/p>\n

We evaluated an early version of Claude Mythos Preview for risk assessment during a limited window in March 2026. We estimated a 50%-time-horizon of at least 16hrs (95% CI 8.5hrs to 55hrs) on our task suite, at the upper end of what we can measure without new tasks. <\/p>\n

<\/p>\n

11:41 PM \u00b7 May 8, 2026 \u00b7 849K Views<\/p>\n

65 Replies \u00b7 234 Reposts \u00b7 2K Likes<\/p>\n

<\/a><\/p>\n

Even the usually sober forecaster Peter Wildeford worried that Mythos had \u201cbroken\u201d the graph, meaning that we could no longer measure the limits of AI capabilities:<\/p>\n

\"\"<\/a><\/p>\n

Hold on. Let\u2019s take a deep breath. <\/p>\n

(And let\u2019s ignore the fact that \u201cDeep learning is hitting a wall\u201d was an essay about the limits of pure scaling,<\/a> rather than what Wildeford is discussing.)<\/p>\n

What the METR \u201ctime horizon\u201d graph is measuring \u2013 with two important asterisks that I will get to \u2014 is the length (measured in time) of software development tasks that frontier models can complete, normed against human software engineers. <\/p>\n

It used to be that the best \u201cfrontier models\u201d could \u201csucceed\u201d at tasks that would occupy humans for a minute, then they could \u201csucceed\u201d at two minute task, then four, then eight etc; it\u2019s up to sixteen hours now (but wait for the asterisks). <\/p>\n

The implication is that systems are steadily getting better and better, at tasks that are more and more complex.<\/p>\n

As Ernest Davis and I discussed a year ago, there are a bunch of problems with how the task is conceived and implemented<\/a>, but for now let\u2019s just stipulate for the sake of argument that the graph has been carefully made.<\/p>\n

Here\u2019s some context<\/a>:<\/p>\n

Claude Code is a real advance; Mythos probably builds on some of what is learned there. But\u2026 <\/p>\n

If you read METR\u2019s tweet about their graph carefully, it is about achieving *50%* success. Not 100 or 99 or even 90. (There is an 80% version, and it looks less ominous; it is the same general shape, but much lower overall performance.)<\/p>\n

For exactly that reason, Wildeford\u2019s concern about Mythos \u201cbreaking the graph\u201d is a red herring. He\u2019s saying that Mythos basically solves all the tasks that METR had prepared for the benchmark. And I don\u2019t doubt that that\u2019s true. But it\u2019s only true at the arbitrary 50% success level. There is plenty of headroom left on the current METR set of tasks if you simply demand 80% success, even more headroom if you demand 95% success. (It\u2019s also stacking the deck to only look at jobs that take a day or two, when the real job of a software engineer may involve overseeing projects that take months or years.)<\/p>\n

More broadly, the key problem with GenAI has been reliability; a graph that demands only 50% success does not address reliable performance. At all. <\/p>\n

The graph pertains only to software-development tasks. Not general intelligence. <\/p>\n

It certainly doesn\u2019t tell you that Mythos can do *most* things that humans can do in 16 hours, let alone do them reliably. (Example, watch a two-hour Hollywood movie that nobody has seen before and sensibly discuss key plot points.)<\/p>\n

Importantly, the graph doesn\u2019t show you *how* the improvements have been made. As noted in my newsletter, a lot of the advance in recent months is likely from the incorporation of symbolic tools (like code interpreters, verification, and harnesses) rather than from model scaling per se. (Incidentally, this is yet another vindication of neurosymbolic AI \u2013 not proof that LLMs themselves can be perpetually scaled. Nor a proof that another trillion dollars will indefinitely continue the trends shown in the graph.)<\/p>\n

Per a graph that Ramez Naam showed a month ago, Mythos is not actually far off trend on the ECI benchmark, which is a broader measure.<\/p>\n

\"X<\/p>\n

Ramez Naam@ramez<\/p>\n

Anthropic’s Mythos does not appear to show any acceleration of ECI. After normalizing Anthropic’s internal ECI with @EpochAIResearch ‘s public ECI, it’s clear that the two metrics are extremely close, and that Mythos is pretty much on trend, just slightly above GPT 5.4. \/1 <\/p>\n

<\/p>\n

6:30 PM \u00b7 Apr 8, 2026 \u00b7 409K Views<\/p>\n

40 Replies \u00b7 101 Reposts \u00b7 917 Likes<\/p>\n

<\/a><\/p>\n

Bottom line: Mythos is awfully good at coding relative to its predecessors, but 50% is a low bar, and (a) we don\u2019t have data at 95% or 99% success, we don\u2019t know that the curves will keep going, and (b) we don\u2019t have evidence that Mythos is actually an important step towards broad superintelligence.<\/p>\n

Instead, its techniques likely work best with things like coding and math, where formal verification (good old symbolic AI for the win!) can straightforwardly apply. <\/p>\n

Ramez Naam was sharp on this point, too, yesterday:<\/p>\n

\"\"<\/a><\/p>\n

\u00a7<\/p>\n

Here\u2019s an even wilder extrapolation from a few days ago, about money rather than task performance:<\/p>\n

\"\"<\/a><\/p>\n

To anticipate that Anthropic will have $2T revenue in 2030 is a perfect example of what I have often called the trillion pound baby fallacy: Just because a baby doubles in weight in its first four months doesn\u2019t mean it will continue to doubling every few months til he goes away to college.<\/p>\n

\"\"<\/a><\/p>\n

\u00a7 <\/p>\n

Over and over I saw variants on the trillion pound baby fallacy yesterday with the METR graph, with people assuming that processes that initially doubled would continue unimpaired, indefinitely. Very few exponential processes do. <\/p>\n

Babies don\u2019t keep doubling forever, and nor will AI progress. We might hit resource constraints (energy, chips etc); \u201cbenchmarkmaxxing\u201d (teaching to the test, which hears means building tools that focus around software design) may have limits; formal verification techinques may hit limits in less formal problems; some types of challanges (e.g., reasoning accurately with respect to world models, reducing hallucinations etc) may simply not be amenable to current approaches; and so on. <\/p>\n

We can be absolutely sure that the task length \u201ctime horizon\u201d for AI is not going to keep doubling until \u201ctime horizons will be 580 times the age of the universe<\/a>\u201d as Lisan al-Ghaib<\/a> joked.<\/p>\n

And most importantly, solving (some aspects of) software design is not open-ended intelligence. AI is definitely getting better at some things, but there is no reason to think that it is close to fully general yet. <\/p>\n

My strong intuition is that Mythos will be under 20% and perhaps under 10% on the Remote Labor Index (a benchmark of percent of online tasks a bot can do), and with no meaningful improvement on doing physical jobs \u2014 which means the number of actual full human jobs that can be entirely replaced will remain small, at least for now.<\/p>\n

In short, there is no need (yet?) to panic. <\/p>\n

Thanks for reading Marcus on AI! If you enjoyed this post, considering sharing it!<\/p>\n

Share<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"1. Take the mostly horizontal part (yellow dots, actually attested); 2. Infer vertical. A couple days ago METR,…\n","protected":false},"author":2,"featured_media":435381,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[345,343,344,85,46,125],"class_list":["post-435380","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-il","tag-israel","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/435380","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/comments?post=435380"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/435380\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media\/435381"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media?parent=435380"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/categories?post=435380"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/tags?post=435380"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}