On-premise medical AI agents for reliable clinical decision-making

Ethics statement

This study utilized deidentified, retrospective clinical data from the MIMIC-IV (version 2.2) and physician-curated cases from previously published reports in VivaBench. The collection of patient information and creation of the MIMIC-IV resource was reviewed and approved by the institutional review boards of the Beth Israel Deaconess Medical Center (BIDMC) and the Massachusetts Institute of Technology (MIT), which granted a waiver of informed consent. No participants were prospectively recruited or compensated for the present study, and no additional informed consent was obtained. All data processing was conducted within a fully on-premise, institutionally governed environment. No protected health information or deidentified clinical text was transmitted to, stored by or accessible to any external entities or model providers. All researchers involved in data analysis of this study completed the required CITI Program training (‘Data or Specimens Only Research’) and adhered strictly to the PhysioNet Credentialed Health Data Use Agreement.

Dataset

To evaluate the agent’s clinical reasoning capabilities across distinct diagnostic settings and to assess generalizability, we used three benchmarks spanning two independent data sources (Fig. 1). Two benchmarks are derived from MIMIC-IV, a publicly available dataset of deidentified EHRs from BIDMC: MIRA-v2 (n = 551; seven conditions), adapted from the MIRA29 framework, which serves as the primary benchmark for diagnostic decision-making, and CDM30 (n = 2,400; four acute abdominal conditions), which serves as a cross-validation benchmark for diagnostic reasoning at scale. A third benchmark, VivaBench32 (n = 990; 10 clinical specialty groups), provides independent external validation across a broad multi-specialty case mix. Each benchmark is described in detail in Extended Data Fig. 1a.

The primary benchmark (MIRA-v2) includes seven conditions, including appendicitis, cholecystitis, diverticulitis, pancreatitis, pneumonia, pulmonary embolism and urinary tract infection. It simulates early emergency assessment. We excluded the pancreatic cancer subset (n = 23) from the original MIRA benchmark for three reasons. First, evaluation of pancreatic masses is inherently multimodal and depends heavily on cross-sectional imaging (computed tomography/magnetic resonance imaging) for diagnosis and staging; because our framework operates on structured text without imaging inputs, including these cases would introduce a modality mismatch that could confound performance and reliability analyses. Second, these encounters more often reflect oncological workup, staging and treatment planning rather than the acute, early-presentation diagnostic reasoning emphasized in this study, shifting the primary decision target from de novo diagnosis toward management decisions. Third, the small sample size limits statistical power and would yield unstable estimates at the disease level.

The cross-validation benchmark (CDM) spans appendicitis, cholecystitis, diverticulitis and pancreatitis, with each case simulating a complete clinical encounter beginning with abdominal pain.

To extend the evaluation beyond the acute-care, single-institution setting represented by the MIMIC-IV-derived benchmarks, we additionally assessed the agent on VivaBench, an independent PubMed-derived diagnostic benchmark (n = 990). VivaBench consists of physician-curated cases drawn from published case reports and spans 10 specialty groups, including cardiovascular, endocrine, gastrointestinal, infectious disease, neurological, hematology/oncology, pediatric and respiratory presentations. Unlike the MIMIC-IV-derived benchmarks, VivaBench is designed to simulate viva-style diagnostic reasoning under uncertainty across broader clinical domains rather than admission-specific emergency encounters. This makes it structurally distinct from the more common-condition, EHR-derived MIMIC benchmarks and less directly comparable in absolute performance.

Agent framework architectureOverview

We developed a fully on-premise, dual-agent simulation framework to evaluate clinical reasoning in autonomous medical agents. The architecture is adapted from the MIRA29 framework and centers on two interacting agents: a Physician Agent and a Patient Agent operating within a controlled local environment. Our implementation prioritizes modularity, observability and operational control and supports future multi-agent extensions. Interoperability and EHR integration standards (for example, FHIR34) were not modeled, as the present study focuses on diagnostic reasoning and confidence signaling rather than deployment integration.

The Patient Agent

The Patient Agent simulates a real patient within the clinical encounter. It is a non-tool-using conversational model restricted to case-specific information provided at initialization, including the chief complaint, a clinical history summary and pre-admission medications when available. For the MIMIC-IV-derived benchmarks, these inputs were derived from the source clinical record; for VivaBench, they were derived from the benchmark-provided case material. Benchmark-specific Patient Agent prompts are provided in Supplementary Tables 25 and 26. Across benchmarks, the Patient Agent was required to remain grounded in the provided case context and to avoid introducing unsupported symptoms, findings, diagnoses or treatments in the baseline setting. Pre-admission medication lists were included for MIRA-v2 when available and omitted for CDM because of data availability.

The Physician Agent

The Physician Agent performs diagnostic reasoning through multi-turn dialogue and tool use, using an on-premise LLM. It can request objective clinical evidence via function calling, including laboratory values, urinalysis, radiology reports, microbiology results and physical examination findings (Extended Data Table 1). Tool outputs are returned as textual observations and appended to the agent’s context for subsequent reasoning. Each episode terminates when the agent invokes a submission action (‘admission’), providing (1) a single diagnosis (Dx) and (2) a structured reasoning trace (R). These two outputs are used to compute the reliability-related metrics described in the ‘Multi-perspective confidence framework’ section. Benchmark-specific Physician Agent prompts are provided in Supplementary Tables 27 and 28.

Planning strategy

Given the context budget constraints of on-premise models, we removed the explicit Plan tool used in the original MIRA framework to reduce prompt overhead and preserve token capacity for dialogue history and tool outputs. The Physician Agent instead performed implicit planning, in which successive tool calls and intermediate reasoning adaptively guided the diagnostic trajectory across turns.

Clinical encounter workflow

Each simulation models a complete clinical encounter from initial presentation to final diagnosis. The encounter opens with the Patient Agent’s initial complaint, instantiated from the case’s chief complaint in the benchmark-specific case material. This opening utterance serves as the first input to the Physician Agent and initiates the diagnostic dialogue.

The core of the encounter is a dynamic, multi-turn interaction. During each encounter, the Physician Agent elicits subjective information from the Patient Agent and retrieves objective findings through the available clinical tools (Extended Data Table 1). Tool outputs are appended to the Physician Agent’s context. The encounter terminates when the Physician Agent invokes the admission tool or reaches the 10-turn limit.

Model selection under on-premise constraints

We screened open-weight LLMs for reliable multi-turn dialogue, structured tool calling and sufficient context length. GLM-4.5-Air35 was selected as the primary model after several candidates were excluded because of malformed tool calls, unsupported serving infrastructure or context-length limitations. During revision, GPT-OSS36, GLM-5 (ref. 37) and Qwen-3.5 (ref. 38) were additionally evaluated. Detailed screening criteria and failure modes are reported in Supplementary Table 24.

Computational cost and resource profiling

We quantified computational cost using token usage and wall clock runtime. Token usage was extracted from run metadata for each completed encounter. Doctor agent token use was analyzed separately from total workflow token use, where total workflow tokens included doctor agent, patient simulator and other workflow component tokens.

For single-run inference, token use was analyzed at the encounter level. For five-run consistency analysis, token use was aggregated at the case level by summing token use across the five repeated encounters for the same case and model temperature configuration. Token use was summarized using median and interquartile range. Five-run overhead ratios were calculated within each model temperature configuration as the median five-run case-level token use divided by the median single-run encounter-level token use.

Wall clock runtime was assessed separately in a controlled latency subset of 35 encounters, consisting of five randomly sampled cases from each of seven disease cohorts. The same subset was used for all four locally hosted model configurations. For each encounter, wall clock time was recorded from simulation start to completion and represents encounter-level runtime, including the doctor agent, patient simulator, tool calls and workflow overhead. Five-run runtime was estimated under a serial execution assumption as five times the measured single-run runtime.

The LLMs were served using vLLM39 on NVIDIA H200 and NVIDIA RTX PRO 6000 GPUs. Model-specific details are in Extended Data Table 2. The system was built using the OpenAI Agents SDK40 to support flexible orchestration of tool use and conversation flow.

Experimental configurationsSingle-agent baseline

Unless otherwise stated, the principal confidence, stress test and selective-autonomy analyses were conducted on MIRA-v2 using a single-agent baseline in which one Physician Agent interacted with one Patient Agent. This baseline used GLM-4.5-Air with T = 0.01, top_p = 1.0 and top_k = 0, where T, top_p and top_k denote sampling temperature, nucleus-sampling probability threshold and top-k sampling cutoff, respectively. We used this configuration for the main reliability analyses to keep the workflow fixed across experiments.

For benchmark−performance comparisons, we evaluated additional on-premise open-weight models under the same agent architecture and evaluation pipeline. On MIRA-v2, these included GLM-4.5-Air, GLM-5, Qwen-3.5 and GPT-OSS; each model is reported at its best-performing temperature setting. On CDM, the main on-premise evaluations were performed with GLM-4.5-Air and Qwen-3.5. Confusion matrix and adjudication analyses in Fig. 2c–f used the GLM-4.5-Air baseline.

Multi-agent extension

To explore potential safety enhancements, we implemented a multi-agent system by introducing a DDx Critic Agent. This agent, powered by a domain-specific LLM (MedGemma), serves as a secondary reviewer. It receives the primary agent’s differential diagnosis and returns a structured critique that highlights weaknesses or potential high-risk alternatives, functioning as an automated ‘devil’s advocate’.

The Physician Agent can optionally consult the Critic Agent before submitting a final diagnosis. The Critic Agent ablation (Extended Data Fig. 2) used the same baseline and differed only by addition of the DDx Critic Agent. The specific instruction governing the Critic Agent is detailed in Supplementary Table 29.

Adversarial stress testing

To assess robustness under informational scarcity, we performed an adversarial stress test comparing the Physician Agent’s performance in the standard, factually grounded simulation (baseline) with a perturbed condition designed to reduce evidentiary grounding in the patient narrative.

In the perturbed condition, we removed the chief complaint and the full history summary (including the appended admission medication section when present) from the Patient Agent’s system prompt, leaving the system prompt with no information about the patient (Supplementary Table 30). Without these structured anchors, the LLM-based Patient Agent was forced to confabulate context, generating a symptom history that was often incomplete, inconsistent or laden with unsupported details relative to the ground truth.

The key manipulation was, therefore, the quality and stability of conversational evidence available to the Physician Agent. This design enabled us to quantify how degraded patient-provided information affects diagnostic accuracy and the agent’s decision-time confidence signals. All other experimental settings were held constant across conditions, including model versions, decoding parameters (for example, temperature), tool availability, Physician Agent prompts and the number of repeated runs per case.

External benchmark configuration

For VivaBench, benchmark-level comparisons were performed with GPT-OSS and Qwen-3.5, both at T = 0.6. The main confidence and triage analyses on VivaBench (Extended Data Fig. 4b–i) used Qwen-3.5 with model-default Hugging Face sampling settings (top_k = 20, top_p = 0.95). Because VivaBench represents viva-style diagnostic encounters rather than admission-specific emergency workflows, benchmark-specific Patient Agent prompts and Physician Agent tool interfaces were used. The core diagnostic reasoning logic, submission format and confidence-estimation procedures remained unchanged.

Sensitivity analyses

Implementation-sensitivity analyses (Extended Data Fig. 5) varied repeated-run count (N = 3, 5, 10), decoding temperature (T = 0.01, 0.3, 0.6, 0.9) and semantic encoder (MiniLM, MPNet and BGE). Except for the repeated-run sensitivity analysis, all confidence, triage and implementation-sensitivity analyses used N = 5 stochastic runs and MiniLM for consistency estimation. In temperature-sensitivity analyses, top_p and top_k were held at the model-specific default settings listed in Extended Data Table 2.

Performance evaluationAutomated benchmark scoring

We evaluated diagnostic performance using benchmark-specific scoring protocols. For the seven-disease MIRA-v2 benchmark, we used an LLM-based adjudication protocol29 (LLM-as-a-judge). The evaluator model was run independently from the clinical agent and received the agent’s predicted diagnosis together with the reference diagnosis label for the case, returning a structured decision on clinical equivalence. All LLM-based adjudication analyses reported here used Gemini 3.1 Flash-Lite41 as the evaluator model. The evaluator prompt used for the MIMIC-IV-derived benchmarks is provided in Supplementary Table 31.

For the four-disease CDM benchmark, we used the published fuzzy-matching evaluation procedure30, which compares the agent’s final diagnosis with a predefined set of permissible terms for each disease class. This protocol was retained to preserve direct comparability with the original CDM leaderboard and previously reported results.

For VivaBench, we also used an LLM-based adjudication procedure but with a benchmark-specific evaluation prompt adapted to the task format (Supplementary Table 32). VivaBench provides both a diagnosis list and an accepted differentials list, whereas the clinical agent outputs a single final diagnosis. We, therefore, aligned the evaluation to the benchmark’s top-1 approximate accuracy principle32: the evaluator assessed the prediction against both the diagnosis list (decision) and the accepted differential list (partial_credit). A case was counted as incorrect only when both decision and partial_credit were false.

To assess agreement between LLM-based adjudication and the published fuzzy-matching protocol in a setting where both were applicable, we performed a head-to-head concordance analysis on the CDM dataset. We evaluated a single prespecified run (GLM-4.5-Air; run 1; pooled n = 2,400). The two methods showed high agreement (Cohen’s κ = 0.95) and did not differ significantly in aggregate accuracy (McNemarʼs test, P > 0.05) (Extended Data Fig. 6), supporting the use of LLM-based adjudication when rigid term list matching is impractical.

Physician evaluation

To validate automated scoring and assess clinical validity beyond administrative labels, we performed blinded physician review in two rounds.

In round 1, we sampled a stratified random subset of 111 MIRA-v2 cases (approximately 20%), broadly reflecting disease prevalence. Two board-certified physicians independently reviewed each case using the same structured clinical information provided to the agent (summarized history, medications, physical examination, laboratory tests, urinalysis, microbiology and radiology). In phase 1, physicians generated an independent diagnosis without access to the dataset label or the agent output. In phase 2, they assessed the clinical validity of both the dataset reference label and the agent diagnosis (each marked clinically valid or not clinically valid). Disagreements were adjudicated by a third board-certified physician, and the final round 1 outcome was determined by majority vote.

Because some low-prevalence conditions remained underrepresented in round 1, particularly pneumonia (n = 6), we conducted a second physician evaluation round to enrich small disease strata. Specifically, we added diverticulitis (+16), pancreatitis (+16) and urinary tract infection (+15) cases to reach approximately 50% physician coverage in those categories and added all remaining pneumonia cases (+23), yielding full physician review for pneumonia (29/29 cases). This produced an additional 70 cases in round 2.

In round 2, six board-certified physicians participated: the three physicians from round 1 and three additional physicians. The 70 cases were randomly partitioned into three fixed subsets: X (n = 20), Y (n = 25) and Z (n = 25). For inter-rater agreement analysis, all six physicians reviewed subset X. The physicians were then split into two groups of three, with one group reviewing X + Y (n = 45) and the other reviewing X + Z (n = 45). Each case was, therefore, independently reviewed by at least three physicians, and round 2 case-level outcomes were determined by majority vote within the corresponding physician set (details in Supplementary Table 3).

Qualitative case review

To characterize failure modes, confidence-state discordance and benchmark-label ambiguity, physicians conducted targeted review of representative cases from both the MIRA-v2 and VivaBench analyses. In MIRA-v2, four prespecified baseline cases were reviewed to represent contrasting confidence profiles: two high-confidence incorrect cases by automated scoring and two low-confidence correct cases. Review focused on missing or misleading evidence, reasoning trajectory and safety-relevant implications for escalation. In VivaBench, we additionally reviewed 10 baseline cases under the Qwen-3.5 configuration that had high ConsistencyDx (≥0.85) but were scored as false by the LLM-based evaluator and, thus, represented cases incorrectly retained in the autonomous stream (‘Autonomous Error’) in the selective-autonomy simulations. Two physicians reviewed these cases to determine whether the evaluator-marked discordances were clinically meaningful errors, whether partial credit was warranted, why the model appeared confident despite disagreement with the benchmark endpoint and what missing or later-available evidence would have changed the answer.

Multi-perspective confidence framework

We defined decisional trust using reference-independent, inference-time reliability measures computed from the agent’s outputs without access to ground truth labels. The framework captures three complementary dimensions: internal likelihood, expressed uncertainty in language and behavioral stability across repeated stochastic runs. These dimensions were selected to capture distinct aspects of model behavior: token-level preference under a given context, linguistic expression of certainty and semantic stability across repeated outputs. The framework was informed by previous work on self-consistency reasoning, semantic uncertainty estimation and uncertainty-aware diagnostic systems26,27,42 but evaluates these signals jointly within an autonomous clinical agent workflow and separately for both the final diagnosis (Dx) and reasoning trace (R). Unless stated otherwise, scores were computed per case. For ProbScore, LingCert and ConceptDensity, run-level values were averaged across runs; Consistency was computed once per case using the full set of runs.

Internal Confidence: probabilistic score (ProbScore)

We used token-level likelihood as a lightweight internal likelihood proxy. For each run, let the output text (Dx or R) contain \(k\) tokens \(\left({t}_{1},\ldots ,{t}_{k}\right)\) generated given the context \(C\). We compute the geometric mean token probability in log space:

$${S}_{\mathrm{prob}}=\exp \left(\frac{1}{k}\sum _{i=1}^{k}\mathrm{log}P({t}_{i}|{t}_{ < i},C)\right)$$

We report this score separately for diagnosis and reasoning as ProbScoreDx and ProbScoreR. Because this metric is derived from token-level generation likelihood within a single run, it may reflect local fluency or model preference without guaranteeing factual correctness or calibration. We, therefore, treated ProbScore as one component of a broader reliability framework rather than as a sufficient confidence measure on its own.

Expressed Confidence: linguistic and content cues (LingCert and ConceptDensity)
Hedging-based linguistic certainty (LingCert)

We treated linguistic hedging as a lightweight proxy for expressed uncertainty in the generated text. This choice was informed by previous work on hedge/speculation cues and uncertainty expression in biomedical and clinical language. Because our aim was a transparent, audit-friendly metric applicable directly to generated outputs, we implemented LingCert as a rule-based cue density score rather than a learned uncertainty detector. Given an output text with \(k\) words and \(h\) matched hedging cues, we compute:

$${S}_{\mathrm{ling}}=1-\frac{h}{k+\epsilon }$$

with \(\epsilon =1\) for numerical stability. Higher values indicate fewer hedging cues and, thus, higher expressed certainty. We compute LingCertDx and LingCertR. The hedging lexicon comprises 35 terms curated from established biomedical uncertainty language resources43,44, covering modal verbs (may, might, could), hedging verbs (suggest, indicate, appear, seem) and probabilistic modifiers (possible, probable, unlikely, uncertain); the full list is provided in Extended Data Table 3.

Clinical concept density (ConceptDensity)

To capture whether the output is dominated by clinically meaningful content, we compute the fraction of valid tokens attributable to core clinical concepts. We process text using a biomedical natural language processing (NLP) pipeline (spaCy + scispaCy) with Unified Medical Language System (UMLS) entity linking45,46,47 and retain only entities whose linked concept semantic types (type unique identifiers (TUIs)) fall within a predefined whitelist of core clinical categories (for example, diseases/syndromes, signs/symptoms, findings, anatomical structures and diagnostic/therapeutic procedures; full list in Extended Data Table 4). Let \(T\) be the set of valid tokens (excluding punctuation and whitespace), and let \({T}_{\mathrm{core}}\subseteq T\) be tokens covered by whitelisted entity spans. We define:

$${S}_{\mathrm{cd}}=\frac{|{T}_{\mathrm{core}}|}{{|T|}}$$

We compute ConceptDensityDx and ConceptDensityR. ConceptDensity was included as an exploratory content structure measure to test whether concept-rich outputs aligned with diagnostic reliability. As reported in the Results, ConceptDensityR did not behave as a positive confidence signal and was, therefore, not interpreted as a standalone reliability metric.

Behavioral Confidence: cross-run stability (Consistency)

We defined Behavioral Confidence as the semantic stability of the model’s output across repeated stochastic runs under identical prompts. This choice was motivated by previous work on self-consistency reasoning and related semantic uncertainty methods26,42, but here the resulting cross-run agreement score was used as an inference-time reliability signal rather than as a decoding or answer-selection rule. For each case, we generated N independent runs and embedded each output (Dx or R) using a sentence encoder (all-MiniLM). We then computed the mean pairwise cosine similarity across runs:

$$\bar{s}=\frac{2}{N(N-1)}\sum _{1\le i < j\le N}\cos ({e}_{i},{e}_{j})$$

To obtain a score bounded in [0,1], we clipped the mean pairwise cosine similarity to the unit interval:

$${S}_{\mathrm{con}}=\min (1,\max (0,\overline{s}))$$

We report ConsistencyDx and ConsistencyR. This approach is related to self-consistency methods and semantic entropy but differs in that it operates on full-output embeddings rather than discrete answer clusters and is applied separately to diagnostic conclusions and reasoning traces within a multi-step agentic workflow. Sensitivity of Consistency to the number of runs (N), embedding model choice and decoding temperature is assessed in the ‘Implementation sensitivity analyses of behavioral consistency’ section.

Unless otherwise stated, confidence analyses used five stochastic runs per case and MiniLM as the default sentence encoder for consistency estimation.

Statistical analysis

Discriminative performance of reliability-related metrics was quantified using ROC analysis and summarized by the AUC. Bootstrapped 95% confidence intervals were computed for all AUC estimates. For the primary comparison between ConsistencyDx and ProbScoreDx, differences between correlated ROC curves were assessed using DeLong’s test; FDR-adjusted P values are reported where applicable. Detailed AUC summaries and confidence intervals are provided in the corresponding supplementary tables. Correlations among reliability-related metrics were assessed using Pearsonʼs correlation coefficients. Differences in score distributions between correct and incorrect cases were assessed using two-sided Mann−Whitney U-tests, with Holm correction applied across the family of metrics compared in the same analysis.

For threshold-based selective-autonomy analyses, retained-set accuracy and coverage were summarized with 95% Wilson confidence intervals. Threshold analyses were treated as descriptive operating characteristic analyses over predefined threshold grids rather than as formal threshold optimization procedures.

For the stress test analyses, baseline and perturbed conditions were paired at the case level (n = 551). Paired differences in diagnostic correctness (binary per case) were assessed using the exact McNemarʼs test, whereas paired differences in continuous confidence metrics were assessed using two-sided Wilcoxon signed-rank tests. Where multiple paired comparisons were performed within the same figure or analysis family, P values were adjusted using the Holm method.

For implementation-sensitivity analyses, paired differences in per-case ConsistencyDx values across repeated-run counts, decoding temperatures or embedding models were assessed using the Friedman test. Threshold-specific differences in binary coverage status across conditions were assessed using Cochran’s Q test. When correctness was compared across conditions within the subset of jointly retained cases, Cochran’s Q test was again used, restricted to the complete paired intersection.

Agreement between evaluation methods on paired case-level outcomes was assessed using Cohen’s κ and McNemarʼs test. Agreement between automated judging and physician adjudication was summarized using Gwet’s AC1, together with overall agreement and discordance counts. All statistical tests were two-sided unless otherwise stated.

Use of AI tools

In accordance with the Committee on Publication Ethics (COPE) position statement of 13 February 2023 (https://publicationethics.org/cope-position-statements/ai-author), the authors hereby disclose the use of the following AI model during the writing of this article: GPT-5.2 (OpenAI) for checking spelling and grammar.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.