{"id":638553,"date":"2026-10-23T00:54:47","date_gmt":"2026-10-23T00:54:47","guid":{"rendered":"https:\/\/www.newsbeep.com\/nz\/638553\/"},"modified":"2026-10-23T00:54:47","modified_gmt":"2026-10-23T00:54:47","slug":"measuring-what-medical-ai-can-actually-do","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/nz\/638553\/","title":{"rendered":"Measuring What Medical AI Can Actually Do"},"content":{"rendered":"<p>We assembled 37 benchmark datasets to cover every subcategory and evaluated the performance of nine frontier models against those benchmarks. We used an LLM jury for scoring against clinician-defined criteria, ensuring that the jury\u2019s agreement with clinicians matched the level of clinicians\u2019 agreement with each other. We examine the \u201cwin rate,\u201d the proportion of pairwise comparisons in which each model achieved superior performance across all benchmarks (possible range: 0\u20131). The results show the highest and lowest win rates of LLM models across the top five categories:<\/p>\n<p>Clinical note generation: 0.74 to 0.85<\/p>\n<p>Patient communication and education: 0.76 to 0.89<\/p>\n<p>Medical research assistance: 0.65 to 0.75<\/p>\n<p>Clinical decision support: 0.63 to 0.77<\/p>\n<p>Administration and workflow: 0.53 to 0.63<\/p>\n<p>Models consistently struggled on clinical decision support and administration and workflow tasks, exactly where health systems actually want the most help to address clinician time and administrative cost. Administrative spending in the U.S. alone exceeds <a href=\"https:\/\/pubmed.ncbi.nlm.nih.gov\/38756977\/\" rel=\"nofollow noopener\" target=\"_blank\">$1 trillion a year<\/a>. The categories with the largest potential return scored the lowest.<\/p>\n<p>Tasks Are Workflows<\/p>\n<p>MedHELM benchmarks grade single responses: A prompt goes in, text comes out, and the text is scored. Real tasks in the two weak categories rarely look like that.<\/p>\n<p>A prior authorization is a good example. Someone reads the order in the EHR (electronic health record), checks the payer\u2019s criteria, pulls supporting documentation, fills out a portal form, attaches files, submits, and records the reference number in the chart. A specialist consult has the same shape: retrieve data across encounters, reason about it, place orders, and document. Grading the text a model would produce at step 3 tells you nothing about whether step 7 happens.<\/p>\n<p>We need a different kind of evaluation for these important categories. Give an agent an environment, a goal, and tools. Let it act. Then check what changed in the environment. That is the design behind PhysicianBench, which explores clinical decision support and execution, and HealthAdminBench, which explores administration and workflow.<\/p>\n<p>The machine learning risk prediction chasm represents the wide gap between a model that makes a good prediction and one that changes what happens to a patient. These benchmarks highlight the generative AI chasm representing the wide gap between what a model says and what a model can actually do.<\/p>\n<p>PhysicianBench and MedAgentBench: Can AI Agents Do the Clinical Work?<\/p>\n<p><a href=\"https:\/\/ai.nejm.org\/doi\/full\/10.1056\/AIdbp2500144\" rel=\"nofollow noopener\" target=\"_blank\">MedAgentBench<\/a> established 300 patient-specific clinically derived tasks across 10 categories, written by human physicians. The tasks are based on realistic profiles of 100 patients in a virtual electronic health record compute environment. When this was released in early 2025, even the best frontier models achieved &lt;70% proper task completion. By 2026, agentic advances are now seeing frontier models with over <a href=\"http:\/\/www.anthropic.com\/news\/healthcare-life-sciences\" rel=\"nofollow noopener\" target=\"_blank\">90% task completion<\/a>.<\/p>\n<p><a href=\"https:\/\/arxiv.org\/abs\/2605.02240\" rel=\"nofollow noopener\" target=\"_blank\">PhysicianBench<\/a> pushes the frontiers further with 100 long-horizon, multi-step tasks adapted from real medical consultations between primary care physicians and subspecialists, across 21 specialties. A separate panel of physicians independently reviewed each task. The tasks are attempted against synthetically reconstructed patient records in an EHR sandbox. An average multi-step task takes 27 tool calls.<\/p>\n<p>Collectively, evaluating the task results in 670 checks in total, covering data retrieval, clinical reasoning, action execution, and documentation. Grading is execution-grounded: Task-specific scripts check whether the right orders exist in the record, with LLM judgments for the reasoning and the note.<\/p>\n<p>Across 12 models, the best (GPT-5.5) completed 46% of tasks on a single attempt. The best open-source model, DeepSeek V4-Pro, reached 19%. Consistency is worse: GPT-5.5 properly executed only 28% of tasks in all three of its runs.<\/p>\n<p>Failure analysis gives us even more insight, as there are still limitations in clinical reasoning that account for about half of the failed checkpoints. The errors are mostly incomplete reasoning and near misses, while outright wrong conclusions are rare. In addition, there is a failure that no past benchmark could catch. Among action-execution failures, 62% for GPT-5.5 and 73% for Claude Opus 4.6 were cases where the agent wrote the correct action in its note and never placed the order in the EHR. The AI generated beautifully written words into a good plan, but never actually got the work done.<\/p>\n<p>HealthAdminBench: Can AI Agents Do the Non-Clinical Work?<\/p>\n<p><a href=\"https:\/\/arxiv.org\/abs\/2604.09937\" rel=\"nofollow noopener\" target=\"_blank\">HealthAdminBench<\/a> targets the bulk of healthcare system effort and staffing: the billing, authorization, and administrative work that happens behind the scenes of every patient encounter. The number of behind-the-scenes administrative staff is <a href=\"https:\/\/hmpi.org\/2026\/07\/10\/the-growth-in-healthcare-administrators-since-2004-in-the-united-states\/\" rel=\"nofollow noopener\" target=\"_blank\">rapidly growing<\/a> and may already <a href=\"https:\/\/www.beckershospitalreview.com\/uncategorized\/10-administrative-workers-for-every-physician-why-healthcare-job-growth-reflects-inefficiency\/\" rel=\"nofollow noopener\" target=\"_blank\">outnumber frontline clinicians 10:1<\/a>. We spent more than 100 hours observing administrators at Stanford Health Care to build four simulated web environments, an EHR, two payer portals, and a fax system. AI agents are able to operate them the way a person would, through screenshots, clicks, and typing.<\/p>\n<p>There are 135 tasks across three workflows (60 prior authorizations, 60 appeals and denials, 15 durable medical equipment orders) at three difficulty levels, designed with practicing administrators. Each task decomposes into subtasks, and 1,698 verifiers grade them: 1,177 deterministic checks and 521 LLM-judge rubrics. All patient data is synthetic, allowing us to release the environments and run a leaderboard for public use.<\/p>\n<p>The best agent tested at the time, Claude Opus 4.6 in its native computer-use mode, completed 36.3% of tasks end-to-end. The best open-weight model, Kimi K2.5, completed 15.6%.<\/p>\n<p>A key distinction is that GPT-5.4 in computer-use mode passed 82.8% of individual subtasks but completed only 26.7% of whole tasks. Competence on the parts does not add up to reliability in a whole workflow. Filling out a form, then sending it to the wrong address doesn\u2019t count. Patients aren\u2019t going to grant partial credit for completing the documentation while prescribing the wrong medicine.<\/p>\n<p>Here, the failures are coordination failures. Agents gather information in the EHR that they will need much later in a payer portal, fail to write it down, and lose it. They miss file uploads and downloads even when the task requires them. They rarely go back to fix an earlier step, and run out of budget instead.<\/p>\n<p>Reading the Benches Together<\/p>\n<p>These benchmarks make complementary bets. PhysicianBench prioritizes fidelity around clinician-facing processes based on real clinical records. HealthAdminBench prioritizes replication of human workflows for extensive administrative processes in healthcare systems.<\/p>\n<p>A health system governing AI adoption needs both kinds of applications, with evidence for reproducibility and transparency marked by release of (synthetic) data, public code, and leaderboards.\u00a0<\/p>\n<p>The headline performance of 46% task completion in PhysicianBench is not directly comparable with 36% task completion in HealthAdminBench. The environments, model sets, interfaces, and grading schemes differ. The defensible claim is directional: Two independently built, execution-grounded benchmarks, one clinical and one administrative, both find today\u2019s best agents finishing less than half of the work. In both, the dominant failures come from failing to carry information across encounters, documents, and systems, which points to agent architecture and memory (i.e., better harnesses) as the fix, at least as much as bigger models.<\/p>\n<p>Both benchmarks trace back to the categories in which MedHELM found the weakest performance. MedHELM (<a href=\"http:\/\/www.medhelm.org\/\" rel=\"nofollow noopener\" target=\"_blank\">www.medhelm.org<\/a>) is community-owned and released under an Apache 2.0 license, and the point of an open framework is that others can contribute category-specific, <a href=\"https:\/\/arxiv.org\/abs\/2608.07796\" rel=\"nofollow noopener\" target=\"_blank\">workflow-level benchmarks<\/a>. In fact, we are contributing both HealthAdminBench and PhysicianBench. If your health system has a workflow that matters and a benchmark to test it, we invite you to contribute it to MedHELM.<\/p>\n","protected":false},"excerpt":{"rendered":"We assembled 37 benchmark datasets to cover every subcategory and evaluated the performance of nine frontier models against&hellip;\n","protected":false},"author":2,"featured_media":638554,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[34],"tags":[134,527,111,139,69],"class_list":["post-638553","post","type-post","status-publish","format-standard","has-post-thumbnail","category-healthcare","tag-health","tag-healthcare","tag-new-zealand","tag-newzealand","tag-nz"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/638553","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=638553"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/638553\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/638554"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=638553"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=638553"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=638553"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}