{"id":173746,"date":"2025-09-22T10:25:15","date_gmt":"2025-09-22T10:25:15","guid":{"rendered":"https:\/\/www.newsbeep.com\/us\/173746\/"},"modified":"2025-09-22T10:25:15","modified_gmt":"2025-09-22T10:25:15","slug":"in-the-loop-or-on-the-loop-the-conundrum-of-ai-clinical-decision-support","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/us\/173746\/","title":{"rendered":"In the Loop or On the Loop: The Conundrum of AI Clinical Decision Support"},"content":{"rendered":"

\"\"<\/a>Opening the two-day 2025 Nudges in Health Care Symposium, M. Kit Delgado, MD, MS<\/a>, LDI Senior Fellow and Director of the Penn Medicine Nudge Unit<\/a>, welcomed more than 300 attendees from 30 health systems, universities, and other health care organizations to the seventh annual symposium. (Photos: Hoag Levins)<\/p>\n

The latest wave of health technology transformation was set in motion by the 2009 HITECH Act, which mandated and incentivized the adoption of electronic health records (EHRs). Now, those rapidly advancing systems are spawning a second revolution\u2014evolving into massive \u201cnudge\u201d engines that weave human clinicians and machine intelligence together in real-time decision-making.<\/p>\n

Rapidly emerging research is demonstrating AI large language models can perform as well or better than clinicians alone or clinicians with AI assistance in medical reasoning and decision making. This new trend raises the stark question for clinicians as to whether they should remain \u201cin the loop,\u201d controlling final decisions, vs. relinquishing some control with some continuous oversight. That tension was at the heart of the discussion at the 2025 Penn Medicine Nudges in Health Care Symposium on how AI can be leveraged to improve clinical decision support.<\/p>\n

Sharing Authority<\/p>\n

The reach of AI is already vast throughout health care, touching logistics, economics, diagnostics, and treatment. But future success, keynote speaker Adam Rodman, MD, MPH<\/a>, argued, will hinge on how willing medical professionals are to share their authority with algorithms that in many domains are already proving more accurate than humans in complex decision-making.<\/p>\n

Rodman, Associate Editor of the New England Journal of Medicine, is widely regarded as a leading voice in the AI and clinical decision support field. He directs AI programs at the Carl J. Shapiro Center for Education and Research, affiliated with Harvard Medical School and Beth Israel Deaconess Medical Center, and is a visiting researcher at Google.<\/p>\n

More Event Photos Below<\/a><\/p>\n

\"\"<\/a>In his keynote speech, Adam Rodman, MD, MPH<\/a>, discussed how AI is rapidly moving from the margins of medicine into the center of clinical decision making, posing stark questions about how much authority physicians may have to give up to optimize AI-driven decision support systems that are more accurate than the doctors are.<\/p>\n

\"\"<\/a>Rodman pointed to the different characteristics of Star Wars AI robots R2-D2 and C-3PO to illustrate the differences between in the loop and on the loop methods of integrating AI systems and clinicians into the decision support process.<\/p>\n

Rodman challenged conventional thinking around AI in healthcare, particularly the assumption that human collaboration\u2014or \u201chumans in the loop\u201d\u2014is inherently better than systems where humans provide oversight from a distance (\u201chumans on the loop\u201d). He argued that human-in-the-loop models can lead to cognitive deskilling, inefficiency, and poor scalability in clinical settings. At the other extreme, fully autonomous AI systems\u2014where humans are entirely out of the loop\u2014raise dystopian concerns reminiscent of science fiction, and are clearly not the solution either. Instead, he advocated for a \u201chuman on the loop\u201d approach: allowing AI to perform tasks it excels at, while humans retain supervisory control.<\/p>\n

He acknowledged that realizing this vision comes with significant socio-technical challenges, including the absence of a robust regulatory framework and unclear payment models. Importantly, he reminded the audience that while AI is powerful, it is not magic\u2014its success depends on thoughtful integration, oversight, and trust.<\/p>\n

Letting AI Run by Itself<\/p>\n

\u201cWhen I ran two randomized controlled trials,\u201d he explained, \u201cI started getting hate mail attacks in my office because I showed again what every single researcher in this field has shown for decades, which is that just giving a human an AI system doesn\u2019t inherently improve that human\u2019s performance, and it doesn\u2019t improve their performance as much as if the AI system had just run by itself. We\u2019ve now seen this in diagnostic reasoning, management reasoning, and in breast, radiology, and chest X-ray areas. We see it in field after field.\u201d<\/p>\n

\u201cGenerative AI is different because it can have interactions with patients. About two years ago, Google developed a system called the Articulate Medical Intelligence Explorer (AMIE) that talks to patients,\u201d Rodman continued. \u201cThis is an evaluation of standardized patients. What they found is that in standardized patients, when it\u2019s blinded between doctors and the AI system, the AI system has equivalent or better diagnostic accuracy. And it\u2019s basically rated better by the patients in every domain.\u201d<\/p>\n

As an example of health systems that are already using AI to interview patients and prepare a summary and initial treatment recommendations for physician review, Rodman cited the K Health system at Cedars-Sinai Los Angeles. It collects patient symptoms, asks follow-up questions, and cross-references the input with electronic health records and data from other cases.<\/p>\n

AI Beats Cedars-Sinai Physicians<\/p>\n

A study published earlier this year in the Annals of Internal Medicine compared the Cedars-Sinai AI treatment recommendations against those of physicians and found that 77 percent of AI recommendations were rated as optimal, while only 67 percent of physician decisions were rated optimal.<\/p>\n

\u201cThis is not science fiction,\u201d said Rodman. \u201cWe are deploying these patient-facing AI systems. The performance paradox is a problem. Scalability is a problem, but the health care system is in a lot of trouble because there are a lot of sick people and not enough doctors.\u201d<\/p>\n

A performance paradox refers to the fact that success in developing AI components for clinical support does not guarantee improved real-world care. The complex play of system integration, compatibility, and clinician interaction is pivotal to ensuring AI achieves its potential benefits in health care settings.<\/p>\n

One of the most famous examples of the performance paradox was the 2009 Air France 447 disaster when the pilots overruled the autopilot that was telling them to level out the Airbus A330. They kept climbing until the plane stalled and crashed into the ocean, killing everyone onboard.<\/p>\n

Overruling the Algorithm<\/p>\n

\u201cWe see this over and over again,\u201d said Rodman. \u201cIn quantitative trading, whenever the traders overrule the algorithm, they actually do worse. We\u2019ve now seen this in chess and Go board games, sentencing support, weather forecasting, and industrial quality control.\u201d<\/p>\n

\u201cOne of the big challenges and worries I have is that we might end up with this really powerful technology that doesn\u2019t make the difference that we want, because we\u2019re not able to move from human in the loop to human on the loop,\u201d said Rodman. \u201cWhat is the biggest problem? Liability. Now, it\u2019s all with the individual provider. There is not currently a system to shift liability. And because of that, companies are not making investments in this technology. It requires partnerships that health systems don\u2019t really allow for a tech company now. But you cannot build a system that goes from human in the loop to human on the loop because you must deploy a human in the loop system first and then iterate to safely get it to human on the loop. <\/p>\n

Behind the Curtain<\/p>\n

\u201cAs I said earlier, tech is remarkably powerful, but it is not magic,\u201d said Rodman. \u201cAnd just because you have a general-purpose diagnostic or measurement system that can outperform humans doesn\u2019t mean it\u2019s going to make anything better, because the challenges are not just technical, they\u2019re sociotechnical. They involve the way that we deliver care. And this is something that I think the people in this symposium and this specialty are very well positioned to figure out. That\u2019s going to take years\u2014a decade of very, very careful and deliberate work and advocacy to make this happen knowing there is no tech miracle. There is no superintelligence hidden behind the curtain like in \u2018The Wizard of Oz,\u2019 right? In order to make this happen, we have to do the work.\u201d<\/p>\n

These are some other scenes from the symposium:<\/p>\n

\"\"<\/a>Raina Merchant, MD, MSHP,<\/a> LDI Senior Fellow, and Vice President and Chief Transformation Office, University of Pennsylvania Health System, noted that: \u201cNine years after Penn Medicine created the first behavioral science \u2018nudge\u2019 unit inside a health system, nudges have become critical to our core business of improving outcomes, reducing burdens, and transforming health care delivery.\u201d<\/p>\n

\"\"<\/a>The two-day event\u2019s presenters discussed the latest insights related to nudge methodologies that could shift the behavior of patients and clinicians in ways that improved outcomes. LDI Senior Fellow and Nudge Unit Director M. Kit Delgado, MD, MS<\/a>, pointed out that one its recent projects changing EHR defaults led to 20 percentage point increase in 90 day prescriptions for statins, which has been shown to increase adherence and decrease mortality..<\/p>\n

\"\"<\/a>Keynote speaker Katherine Milkman, PhD,<\/a> LDI Senior Fellow, Wharton School Professor, and Co-Director of the Behavior Change for Good Initiative<\/a>, discussed why large-scale mega studies \u2014 coordinated investigations testing many interventions simultaneously \u2014 are essential for accelerating learning, improving outcomes, and avoiding bias in evidence.<\/p>\n

\"\"<\/a>In the \u201cNudging Patients to Better Outcomes\u201d panel, Kimberly Waddell, PhD, MSCI,<\/a> LDI Senior Fellow and Perelman School Assistant Professor, panelists from Permanente Medical Group, Geisinger, Penn Medicine, UCLA, and Ascension presented large scale RCTs of interventions to engage patients. Some studies included sample sizes in the millions of patients.<\/p>\n

\"\"<\/a>LDI Senior Fellow Kimberly Wadell presents an study showing that pended orders and patient text-messaging nudge significantly increased mammogram completion. Other panelist shared their work on testing human-versus-electronic prompts, multichannel EHR\/email nudges for safer diabetes management, and real-time prescription benefit alerts that track patient-specific drug costs.<\/p>\n

\"\"<\/a>LDI Senior Fellow Meeta Kerlin, MD, MSCE<\/a>, an Associate Professor of Pulmonary, Allergy, and Critical Care at the Perelman School, presented on a clinical nudge designed to promote utilization of low tidal volume ventilation (LTVV). LTVV refers to lowering the amount of air provided to minimize injury from lung stretching during mechanical ventilation.<\/p>\n

\"\"<\/a>Sunita Desai, PhD<\/a>, a former Associate LDI Fellow and now an Assistant Professor of Population Health at NYU Langone Health, discussed how a real-time prescription benefits (RTPB) nudge led to reductions in patient out-of-pocket costs for ordered medications, with as much as a 40 percent reduction for high-cost drug classes.<\/p>\n

\"\"<\/a>The AI\/EHR panel, moderated by LDI Senior Fellow and Penn Medicine Chief Health Information Officer, Srinath Adusumalli, MD, MSHP,<\/a> emphasized that embedding AI into EHRs like Epic requires balancing innovation with safety and trust while leveraging new platforms for predictive care, ensuring transparency and customizability for clinicians, and addressing alert fatigue and bias.<\/p>\n

\"\"<\/a>Moderator Adusumalli emphasized the critical questions raised by the rapid integration of AI into clinical decision support. \u201cHow do we assess the safety, utility, and effectiveness of these tools? What standards do we compare them to? How do we best design our user interactions with them? What are the medical, legal, regulatory, and compliance implications, and how should the care delivered with them be reimbursed?\u201d he asked.<\/p>\n

\"\"<\/a>Panelist Carissa Kathuria, BSE,<\/a> R&D Group Lead at Epic Systems Corporation, described the company\u2019s massive new community-based data platform called Cosmos<\/a>, that contains 200 million de-identified patient records. It is being paired with new architecture to create predictive models that can anticipate a patient\u2019s \u201cnext event,\u201d much the same as large language models anticipate the next word.<\/p>\n

\"\"<\/a>The Symposium\u2019s poster sessions featured 48 posters across a broad mix of nudge-related studies ranging from \u201cTransforming Hypertension Management in Primary Care,\u201d and \u201cRelief Is Just a Text Message Away: Gastrointestinal Symptom Check-In Post Living Liver Donation,\u201d to \u201cThe Experience of African American Adults with Low Health Literacy When Accessing Health Care,\u201d and \u201cReducing Patient Falls in Community Radiology.\u201d<\/p>\n

\"\"<\/a>Ashley West, PhD<\/a>, Director of Behavioral Design at Lirio (left), discusses her poster, \u201cDesigning an \u2018At Home\u2019 Digital Health Intervention for Supporting Chronic Condition Lifestyle Management,\u201d with LDI Senior Fellow Ren\u00e9e Betancourt, MD<\/a>, an Associate Professor of Clinical Family Medicine and Community Health at the Perelman School of Medicine.<\/p>\n

Author<\/p>\n

\"\"<\/p>\n

More LDI News<\/p>\n

\t\t\t\"\"<\/p>\n

\n\t\t\t\t\t\tIn Their Own Words\n\t\t\t\t\t<\/p>\n

\n\t\t\t\t\t\tPopulation Health\n\t\t\t\t\t<\/p>\n

\t\t\t\t\tWhy RFK Jr.\u2019s Public Health Agenda Threatens Us All<\/a><\/p>\n

Former Philadelphia Health Commissioner Warns That Gutting the CDC, Undermining Vaccines, and Politicizing Science Will Leave the U.S. Dangerously Unprepared for the Next Pandemic<\/p>\n

\t\t\t\t\t\t\tSeptember 19, 2025<\/p>\n

\t\t\t\"\"<\/p>\n

\n\t\t\t\t\t\tBlog Post\n\t\t\t\t\t<\/p>\n

\n\t\t\t\t\t\tHealth Care Payment and Financing\n\t\t\t\t\t<\/p>\n

\t\t\t\t\tStudy Finds Hospice Providers \u201cGame\u201d Medicare Payment Cap, But With Modest Impact on Costs<\/a><\/p>\n

Some Hospice Agencies Adjusted Admissions and Discharges to Maximize Payments<\/p>\n

\t\t\t\t\t\t\tSeptember 16, 2025<\/p>\n

\t\t\t\"\"<\/p>\n

\n\t\t\t\t\t\tImproving Care for Older Adults\n\t\t\t\t\t<\/p>\n

\t\t\t\t\tUniversity of Pennsylvania Teams Up with CenterWell to Help Advance Primary Care Excellence and Access for Seniors<\/a><\/p>\n

New Program Launches Research Initiatives Focused on Improving Primary Care for Older Adults<\/p>\n

\t\t\t\t\t\t\tSeptember 15, 2025<\/p>\n

\t\t\t\"Mother<\/p>\n

\n\t\t\t\t\t\tBlog Post\n\t\t\t\t\t<\/p>\n

\n\t\t\t\t\t\tPopulation Health\n\t\t\t\t\t<\/p>\n

\t\t\t\t\tNew Parents Put Infants\u2019 Health First, While Their Own Suffers<\/a><\/p>\n

Rural Parents Had More Emergency Visits and Insurance Loss Than Urban Peers, an LDI Study Shows. Integrated Baby Visits Could Help All Parents Be Healthier<\/p>\n

\t\t\t\t\t\t\tSeptember 10, 2025<\/p>\n

\t\t\t\"\"<\/p>\n

\n\t\t\t\t\t\tNews\n\t\t\t\t\t<\/p>\n

\n\t\t\t\t\t\tSubstance Use Disorder\n\t\t\t\t\t<\/p>\n

\t\t\t\t\tCHERISH Research Center\u2019s Grant Is Renewed<\/a><\/p>\n

Penn and Four Other Partners Focus on the Health Economics of Substance Use Disorder<\/p>\n

\t\t\t\t\t\t\tSeptember 10, 2025<\/p>\n

\t\t\t\"stethoscope<\/p>\n

\n\t\t\t\t\t\tIn Their Own Words\n\t\t\t\t\t<\/p>\n

\n\t\t\t\t\t\tHealth Equity\n\t\t\t\t\t<\/p>\n

\t\t\t\t\tSafeguarding Nursing Science Amid Political Retrenchment<\/a><\/p>\n

Experts Say Nursing Ethics Can Help Researchers Confront Federal Disinvestment, Defend Science, and Advance Health Equity<\/p>\n

\t\t\t\t\t\t\tSeptember 9, 2025<\/p>\n","protected":false},"excerpt":{"rendered":"Opening the two-day 2025 Nudges in Health Care Symposium, M. Kit Delgado, MD, MS, LDI Senior Fellow and…\n","protected":false},"author":2,"featured_media":173747,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45],"tags":[182,181,507,74],"class_list":{"0":"post-173746","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/173746","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/comments?post=173746"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/173746\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media\/173747"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media?parent=173746"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/categories?post=173746"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/tags?post=173746"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}