{"id":519779,"date":"2026-03-12T20:27:23","date_gmt":"2026-03-12T20:27:23","guid":{"rendered":"https:\/\/www.newsbeep.com\/us\/519779\/"},"modified":"2026-03-12T20:27:23","modified_gmt":"2026-03-12T20:27:23","slug":"a-clinical-environment-simulator-for-dynamic-ai-evaluation","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/us\/519779\/","title":{"rendered":"A clinical environment simulator for dynamic AI evaluation"},"content":{"rendered":"

Goh, E. et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw. Open 7, e2440969 (2024).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

McDuff, D. et al. Towards accurate differential diagnosis with large language models. Nature 642, 451\u2013457 (2025).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Cabral, S. et al. Clinical reasoning of a generative artificial intelligence model compared with physicians. JAMA Intern. Med. 184, 581\u2013583 (2024).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Goh, E. et al. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat. Med. 31, 1233\u20131238 (2025).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Tu, T. et al. Towards conversational diagnostic artificial intelligence. Nature 642, 442\u2013450 (2025).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Gao, S. et al. TxAgent: an AI agent for therapeutic reasoning across a universe of tools. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2503.10970<\/a> (2025).<\/p>\n

Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172\u2013180 (2023).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Sandmann, S. et al. Benchmark evaluation of DeepSeek large language models in clinical decision-making. Nat. Med. 31, 2546\u20132549 (2025).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Tordjman, M. et al. Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning. Nat. Med. 31, 2550\u20132555 (2025).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Liu, X. et al. A generalist medical language model for disease diagnosis assistance. Nat. Med. 31, 932\u2013942 (2025).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning 248\u2013260 (PMLR, 2022).<\/p>\n

Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X. PubMedQA: a dataset for biomedical research question answering. In Proc. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (eds Inui, K. et al.) 2567\u20132577 (ACL, 2019).<\/p>\n

Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. 11, 6421 (2021).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Schmidgall, S. et al. AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2405.07960<\/a> (2024).<\/p>\n

Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30, 2613\u20132622 (2024).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Fan, Z. et al. AI Hospital: benchmarking large language models in a multi-agent medical interaction simulator. In Proc. 31st International Conference on Computational Linguistics 10183\u201310213 (ACL, 2025).<\/p>\n

Li, J. et al. Agent Hospital: a simulacrum of hospital with evolvable medical agents. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2405.02957<\/a> (2024).<\/p>\n

Bedi, S. et al. Holistic evaluation of large language models for medical tasks with MedHELM. Nat. Med. https:\/\/doi.org\/10.1038\/s41591-025-04151-2<\/a> (2026).<\/p>\n

Zhang, S. et al. Rethinking human-AI collaboration in complex medical decision making: a case study in sepsis diagnosis. In Proc. 2024 CHI Conference on Human Factors in Computing Systems 445, 1\u201318 (ACM, 2024).<\/p>\n

Nori, H. et al. Sequential diagnosis with language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2506.22405<\/a> (2025).<\/p>\n

Bedi, S., Mlauzi, I., Shin, D., Koyejo, S. & Shah, N. H. The optimization paradox in clinical AI multi-agent systems. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2506.06574<\/a> (2025).<\/p>\n

Rosenthal, J. T., Beecy, A. & Sabuncu, M. R. Rethinking clinical trials for medical AI with dynamic deployments of adaptive systems. NPJ Digit. Med. 8, 252 (2025).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Palepu, A. et al. Towards conversational AI for disease management. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2503.06074<\/a> (2025).<\/p>\n

Johnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci. Data 3, 160035 (2016).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Johnson, A. E. W. et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci. Data 10, 1 (2023).<\/p>\n

Kansal, A., Chen, E., Jin, B. T., Rajpurkar, P. & Kim, D. A. MC-MED, multimodal clinical monitoring in the emergency department. Sci. Data 12, 1094 (2025).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Lazic, D. A., Grujic, V. & Tanaskovic, M. The role of flight simulation in flight training of pilots for crisis management. SFJD 3, 3624\u20133636 (2022).<\/p>\n

Article<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Allerton, D. J. The impact of flight simulation in aerospace. Aeronaut. J. 114, 747\u2013756 (2010).<\/p>\n

Article<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Mahmood, F. A benchmarking crisis in biomedical machine learning. Nat. Med. 31, 1060 (2025).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354\u2013359 (2017).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484\u2013489 (2016).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Page, B., Irving, D., Amalberti, R. & Vincent, C. Health services under pressure: a scoping review and development of a taxonomy of adaptive strategies. BMJ Qual. Saf. 33, 738\u2013747 (2024).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Morley, C., Unwin, M., Peterson, G. M., Stankovich, J. & Kinsman, L. Emergency department crowding: a systematic review of causes, consequences and solutions. PLoS ONE 13, e0203316 (2018).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Pines, J. M. et al. The impact of emergency department crowding measures on time to antibiotics for patients with community-acquired pneumonia. Ann. Emerg. Med. 50, 510\u2013516 (2007).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Bernstein, S. L. et al. The effect of emergency department crowding on clinically oriented outcomes. Acad. Emerg. Med. 16, 1\u201310 (2009).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Emanuel, E. J. et al. Fair allocation of scarce medical resources in the time of Covid-19. N. Engl. J. Med. 382, 2049\u20132055 (2020).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Johri, S. et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat. Med. 31, 77\u201386 (2025).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Arora, R. K. et al. HealthBench: Evaluating large language models towards improved human health. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2505.08775<\/a> (2025).<\/p>\n

Jiang, Y. et al. MedAgentBench: a virtual EHR environment to benchmark medical LLM agents. NEJM AI 2, 9 (2025).<\/p>\n

Zhang, C. et al. API agents vs. GUI agents: divergence and convergence. In ICML 2025 Workshop on Computer Use Agents (ICML, 2025).<\/p>\n

Finlayson, S. G. et al. Adversarial attacks on medical machine learning. Science 363, 1287\u20131289 (2019).<\/p>\n

Article<\/a>\u00a0
\n CAS<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Javed, H., El-Sappagh, S. & Abuhmed, T. Robustness in deep learning models for medical diagnostics: security and adversarial challenges towards robust AI applications. Artif. Intell. Rev. 58, 12 (2024).<\/p>\n

Kumar, A. et al. OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases. J. Am. Med. Inform. Assoc. 27, 1850\u20131859 (2020).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Elendu, C. et al. The impact of simulation-based training in medical education: a review. Medicine 103, e38813 (2024).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n PubMed Central<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Sinsky, C. et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann. Intern. Med. 165, 753\u2013760 (2016).<\/p>\n

Article<\/a>\u00a0
\n PubMed<\/a>\u00a0
\n
\n Google Scholar<\/a>\u00a0\n <\/p>\n

Tierney, A. A. et al. Ambient artificial intelligence scribes: Learnings after 1 year and over 2.5 million uses. NEJM Catal. Innov. Care Deliv. https:\/\/doi.org\/10.1056\/CAT.25.0040<\/a> (2025).<\/p>\n","protected":false},"excerpt":{"rendered":"Goh, E. et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw. Open…\n","protected":false},"author":2,"featured_media":519780,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[59],"tags":[258,8869,8006,257,97,252,253,8871,8870,8872,8873],"class_list":{"0":"post-519779","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-health-care","8":"tag-biomedicine","9":"tag-cancer-research","10":"tag-diseases","11":"tag-general","12":"tag-health","13":"tag-health-care","14":"tag-healthcare","15":"tag-infectious-diseases","16":"tag-metabolic-diseases","17":"tag-molecular-medicine","18":"tag-neurosciences"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/519779","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/comments?post=519779"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/519779\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media\/519780"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media?parent=519779"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/categories?post=519779"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/tags?post=519779"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}