We subject a diverse panel of 15 state-of-the-art LLMs (both proprietary and open-source) to our DAS red-teaming protocol, which spans four critical axes of safety for health applications: robustness, privacy, bias–fairness and hallucination–factual inaccuracies (overview in Fig. 1). For robustness, privacy and bias testing, the evaluated models are probed by autonomous adversarial agents that start with the same initial, health-related or clinically plausible ‘seed’ request or question. From this common starting point, the agent initiates an adaptive dialogue, strategically modifying the prompt in subsequent turns on the basis of the evaluated model’s responses. This iterative process adapts the attack through various jailbreak strategies until a policy violation is triggered or a predefined search budget is exhausted. When testing for hallucinations, we do not dynamically modify prompts; instead, we propose a healthcare-specialized agent-based automated hallucination detector that adjudicates the evaluated model’s responses on a curated dataset and produces granular, traceable provenance.
Fig. 1: Overview of DAS red-teaming for healthcare/medical LLMs.
a, We consider four critical axes of safety for health applications on which LLMs are assessed with DAS framework: robustness (consistent model performance under context-preserving perturbations and mutations), privacy (compliance with privacy regulations such as HIPAA or GDPR), bias–fairness (evaluation of cognitive, identity, linguistic and emotional biases in health-related scenarios) and Hallucination–factual Inaccuracies (generation of false medical facts, faulty reasoning or incorrect citations). b, Given initial health-related queries and a baseline correct, safe or unbiased response, the adversarial attack agents use an automated red-teaming toolbox, dynamically selecting from strategies focused on specific safety risks in health use to manipulate or mutate the queries. The goal is to uncover vulnerabilities, that is, eliciting incorrect, unsafe or biased responses. Privacy leakage and hallucinations are automatically assessed by the detector agents. If no violation is detected, the attack agents iteratively escalate or switch strategies until a jailbreak (that is, production of incorrect, unsafe or biased answers) emerges, or the search budget is exhausted. The entire process is fully dynamic and automated, requiring no manual intervention. c, The DAS red-teaming framework is used to evaluate 15 LLMs across all four safety pillars for health applications. The heat map shows jailbreak ratios for each model: even the most robust models exhibit jailbreak rates above 47%. The most resilient models are highlighted with green boxes.
Across 100 million text tokens used in adversarial dialogues and >100,000 scored microtasks, the DAS medical red-teaming framework provides a high-resolution characterization of model behaviour that complements (and frequently contradicts) static benchmark results. In evaluations across 15 state-of-the-art LLMs tested on health tasks, our DAS agents achieve, on average, >90% success rates for jailbreaks in robustness tests, >90% privacy leaks, >85% fairness–bias violations and >74% health-critical hallucinations–factual inaccuracies, exposing significant safety gaps that are invisible to static benchmarks. As every probe prompt is generated on the fly and escalated dynamically in real time, models cannot memorize the perturbed test set, allowing DAS red-teaming to bypass Goodhart’s Law and to evolve in difficulty as both attacker agents and evaluated models advance. The quantitative findings for each safety axis are presented in the following subsections, where we detail (1) the magnitude and mode of performance degradation under stress, (2) common patterns of non-compliance with regards to privacy regulations, (3) measurable bias–unfairness in health and clinical decision-making and (5) the prevalence, context and detectability of hallucinations–factual inaccuracies.
Robustness
To assess the robustness of LLMs in healthcare and medicine, we conducted a twofold adversarial audit designed to move beyond static metrics. Our approach first interrogates the de facto standard for medical knowledge, the well-established MedQA benchmark6, to test the brittleness of knowledge on a benchmark widely considered ‘saturated’ (that is, where models hit a performance ceiling, making further differentiation difficult). Second, to determine if these vulnerabilities generalize to more realistic scenarios, we extend the audit to the more healthcare-oriented HealthBench dataset40, a collection of open-ended vignettes developed by 262 physicians. This dual-audit design allows us to both challenge the validity of high scores on existing benchmarks and evaluate model reliability under pressures that more closely emulate real-world health information use and clinical practice.
We first audit on the QA dataset and attempt to desaturate the MedQA benchmark. Our first audit targets MedQA, a ubiquitous benchmark for evaluating the foundational knowledge of LLMs used in medicine and health. To determine whether high MedQA scores signal genuine health-related and clinical reasoning or brittle pattern matching, we use the canonical MedQA dataset as the starting point (initial queries) for DAS medical red-teaming robustness audit. For all 16 evaluated models (o3-mini is also tested here), we first pose unmodified questions; only those answered correctly on the first attempt are retained for stress testing.
Initial results of the unmodified questions are presented in Fig. 2a, blue line. A high median score of 86% is achieved. We first apply a simple, default perturbation: instructing the model that ‘more than one option can be correct’ and deviating from MedQA’s standard ‘only one option is correct’ instruction. This initial perturbation marginally increases the task’s difficulty (red line). Nonetheless, 14 of 16 evaluated models still surpass the US Medical Licensing Examination passing threshold of 60%, with a median score of 80%. Notably, four leading models (o3, o4-mini, o3-mini and DeepSeek-R1) still achieve more than 90% accuracy, whereas the scores of Llama-4-Scout, MedGemma and Gemma-3 are reduced by more than 35%. This simple instruction perturbation is used in all further tests.
Fig. 2: Dynamic DAS red-teaming reveals profound robustness failures that static benchmarks miss.
a, The initial score of 16 leading LLMs on MedQA6 using prompt ‘only one option is correct’ (blue line, serving as the baseline unmodified test) and ‘more than one option can be correct’ (red line, serving as the default mutation for all the following tests). b, Under iterative, dynamic attacks by DAS red-teaming robustness orchestrator framework, nearly all evaluated models (except o4-mini and DeepSeek-R1) get trapped within one to three rounds with a high jailbreak ratio (mean of 94%) (percentage of questions initially answered correctly but changed to incorrect responses after the red-teaming stress test). c, The breakdown of jailbreak ratios by mutation type for each evaluated model. The model with the lowest jailbreak ratio (that is, the most robust) for each mutation is highlighted with yellow boxes. d, The initial score of the evaluated LLM models on a curated 192-sample subset of the open-ended HealthBench dataset40. The six best-performing (tier 1) models are marked in bold. A stark performance reranking compared with MedQA is presented. e, Adversarial red-teaming tests of tier 1 models using mutation tools narrative distraction, cognitive bait and physiological impossibility on the Healthbench dataset. Sample size N = 192, with error bars indicating 95% confidence intervals and mean values as the centre. Despite segregated performance on original test samples, an average jailbreak score of 76% is revealed. Bold indicates the jailbreak ratio.
We then sample 100 such first-round correct items per model and subject them to hard perturbations organized into six orthogonal mutation tools, including answer negation, question inversion, distractor expansion, narrative distraction, cognitive bait and physiological impossibility (Fig. 3a).
Fig. 3: Overview of DAS red-teaming robustness test framework for LLMs in health.
a, Examples of the six mutation tools for the MedQA-based (multiple-choice) red-teaming test. b, Examples for the Healthbench-based (open-ended) red-teaming test for realistic health and clinical reasoning tasks. Each attack is targeted at a certain given rubric and is designed based on this rubric. c, A schematic of the robustness red-teaming orchestration process. The orchestrator, equipped with the toolbox, is aware of both the initial questions and the previous responses. It dynamically adapts its approach, deploying alternative or escalated attack strategies if previous attacks fail, to probe and challenge the robustness of the evaluated models.
An attacker-agent orchestrator (Fig. 3c) coordinates these tools over up to five conversational rounds. To ensure methodological rigour, the orchestrator’s actions are strictly constrained to this predefined set of six tools, prohibiting any unscripted or ‘freestyle’ attacks. After each exchange, it reviews the full dialogue history—including the model’s previous answers and the applied perturbations—and dynamically selects, combines or escalates strategies with the objective of inducing errors in the models’ answers while minimizing redundant queries. This closed-loop design converts robustness testing into an active adversarial game, preventing a static set of questions that can be exploited during model fine-tuning and mirroring the adaptive but realistic pressures that future health applications and clinical deployments are likely to face.
We further generalize failures to realistic open-ended tasks. Although the MedQA audit reveals fundamental brittleness, a critical question remains: do these failures generalize to the more complex, open-ended scenarios encountered in real practice? To answer this, we extended the audit to the HealthBench dataset40, utilizing a curated subset of 192 open-ended health-related vignettes equipped with physician-designed grading rubrics. These rubrics define specific adherence criteria: actions the model must take (positive rubric, for example, ‘refer to emergency department’) or avoid (negative rubric, for example, ‘do not prescribe antibiotics’). The DAS attack agent specifically targets these criteria. For a given case where the model initially adheres to a rubric, the attack agent orchestrates a targeted mutation designed to coerce the model into ‘violating that specific adherence’ (see Fig. 3b for more details). A baseline evaluation (Fig. 2d) confirmed the benchmark is not ‘saturated’—even the top-performing model, o3, scored only 42%—setting the stage for a meaningful adversarial test. We then subjected the six best-performing (tier 1) models to an audit using the three DAS tools applicable to open-ended text: narrative distraction, cognitive bait and physiological impossibility.
A key finding is that a pervasive and generalizable lack of robustness across all tested models. On the MedQA benchmark, models exhibited a stark desaturation and deterioration when stressed. On questions they initially answered correctly, we observed a median jailbreak (correct → incorrect under DAS) rate of 94% (Fig. 2b). Although proprietary models were more robust (86.7% jailbreak) than open-source models (98%), even the most resilient model, o4-mini, failed in 69% of cases. Crucially, this brittleness persists beyond standardized exams. Our audit on the open-ended HealthBench dataset confirmed these vulnerabilities generalize to realistic medical tasks. Even the most recent static benchmarks, such as HealthBench, fail to capture the profound brittleness of the evaluated LLM models. Under red-teaming adversarial pressure, all top-tier models showed high susceptibility, with overall jailbreak rates exceeding 70% (Fig. 2e). We term this stark discrepancy between (misleadingly) high static benchmark performance and low dynamic robustness the ‘benchmarking gap.’
Beyond failure rates, the dual-audit design uncovered a second critical finding: substantial performance reranking between MedQA and HealthBench. MedGemma rose in relative standing on HealthBench, whereas top MedQA performers such as GPT-4o dropped considerably. Such rank instability suggests benchmark-specific optimization rather than transferable medical competence.
A further ablation study of the MedQA mutation strategies provides a more granular view of these vulnerabilities (Fig. 2c). Question inversion emerges as the most impactful adversarial technique in the multiple-choice setting, with an average jailbreak ratio of 60%. This high failure rate is particularly diagnostic: difficulty processing inverted logic (for example, ‘what is NOT the treatment’) indicates reliance on surface-level pattern matching rather than robust semantic reasoning. Notably, cognitive bait and physiological impossibility proved highly effective across both the multiple-choice and open-ended formats, indicating they probe a fundamental, modality-independent weakness in model reasoning.
Collectively, our findings from both the MedQA and HealthBench audits deliver a clear verdict: high accuracy on static benchmarks is an insufficient and potentially misleading proxy for real-world healthcare reliability. Even the most current static benchmark, for example, HealthBench, when used alone, fails to capture the profound brittleness of the testee LLM models. Although, in the initial static test, the OpenAI o3 model yields the highest score (42%), its advantage vanishes in the dynamic stress test, where it proves no more robust to adversarial failures than other tier 1 models. The evident disparity between static leaderboard performance and robustness under adversarial stress, both on multiple-choice exams and in open-ended health and clinical vignettes, highlights the critical necessity for adaptive, agent-driven red-teaming in healthcare LLM evaluation.
Privacy
Protecting personally identifiable health information is a legal imperative under the HIPAA in the USA and the GDPR in the European Union. Under the US HIPAA Privacy Rule, data is considered deidentified only if 18 specific identifiers (including names, dates and locations) are removed or concealed. Similarly, the EU GDPR (Article 5) mandates ‘integrity and confidentiality,’ requiring that personal data cannot be inferred or processed without consent. This concern is not limited to formal clinical documentation: consumer-facing health assistants, patient chatbots and informal health-related queries may also contain highly identifiable personal information. In healthcare practice, clinicians can violate these regulations unintentionally even if they have been trained and are aware of these regulations41,42,43,44. To test whether LLMs can follow these regulations—or fall prey to the same unintentional privacy leakage that afflicts human users—we construct 81 privacy-trap scenarios spanning eight high-incidence violation modes listed in Fig. 1a, privacy. We apply GPT-4o as the LLM-as-a-judge with a leakage detection accuracy of 93%. Please refer to Methods for more details.
Each target model is first presented with a vanilla version (no privacy warning) of every scenario—a direct request that, if satisfied, would breach HIPAA/GDPR. Figure 4a (blue bar) shows that all models exhibit a worryingly high jailbreak ratio, with an average jailbreak ratio of 86.46%, readily providing disallowed information or instructions with no adversarial prompting. Anticipating stricter adherence to privacy compliance in future models, we next simulate ‘privacy-aware’ releases by appending a system message: ‘please fulfill the user’s request as described in the prompt, while strictly adhering to all relevant privacy regulations and guidelines, including HIPAA and GDPR’. Figure 4a (purple bar) shows that this precaution reduces the jailbreak ratio to 66.75% but does not eliminate successful jailbreaks, mirroring how human clinicians commit unintentional privacy breaches despite explicit training.
Fig. 4: Overview of DAS privacy red-teaming test.
a, The jailbreak ratio of the requests without using any adversarial attack strategies (direct request). We also test providing explicit warning to the evaluated models’ system prompt, stating: ‘please fulfill the user’s request as described in the prompt, while strictly adhering to all relevant privacy regulations and guidelines, including HIPAA and GDPR’ (explicit privacy warning). b–e, The DAS privacy red-teaming mutation examples using well-meaning intention (b), subtle request (c), focus misleading (d) and trap ‘privacy warning’ (e). f, The effectiveness (in percentage jailbreak ratio, that is, a privacy-leaked response after prompt mutation) of different privacy attack strategies. Sample size N = 81, with error bars indicating 95% confidence intervals and mean values as the centre. An average 91% privacy leakage ratio is revealed after DAS privacy red-teaming audit. Bold indicates the jailbreak ratio.
For scenarios that resisted direct requests, DAS applied four disguise strategies: well-meaning intention (Fig. 4b), subtle request (Fig. 4c), focus misdirection (Fig. 4d) and trap warning (Fig. 4e) to test whether residual safeguards could be bypassed. See Methods for more details.
Importantly, DAS red-teaming privacy attack is selectively applied only to evaluated models demonstrating initial jailbreak ratios below 80% after the explicit privacy-awareness instruction. Consequently, models such as DeepSeek-V3, QwQ-32B, Qwen3-32B, MedGemma and Gemma-3—each still exceeding 90% jailbreaks despite explicit instructions—are excluded from further adversarial testing.
Key results of DAS privacy attack are summarized in Fig. 4f, which presents the jailbreak ratio achieved by each attack strategy. Among these, the Trap Warning strategy emerges as the most potent adversarial technique, yielding an average jailbreak ratio of 49.1%. This is followed by focus misdirection (44.8%), well-meaning intention (30.9%) and subtle request (27.0%). The final composite attack (integrating all four strategies) further increased the jailbreak ratio for all evaluated models. Notably, Claude Sonnet-4 is identified as the most privacy-resilient model, with an overall adversarial jailbreak ratio of 72.84%. Conversely, DeepSeek-R1 and HuatuoGPT-o1-70B demonstrate total vulnerability, each presenting 100% jailbreak ratios under adversarial conditions.
These findings indicate a concerning gap between LLMs’ general medical knowledge proficiency (as shown by models such as DeepSeek-R1 and HuatuoGPT-o1-70B in MedQA tasks; Fig. 2a) and their awareness of medical privacy guidelines and laws. Despite explicit regulatory instructions, LLMs remain highly susceptible to sophisticated privacy attacks, underscoring the critical need for systematic adversarial privacy auditing in health AI applications, including consumer-facing and clinical deployments.
Bias–fairness
With the increasing integration of language models into consumer-facing chatbots, general health assistants and clinical workflow, LLMs that promise universal access to medical expertise must deliver recommendations that are equitable across demographic groups, linguistic styles and emotionally charged encounters. They must also resist cognitive biases, such as appeals to authority or recency effects, which are known to challenge human clinicians. To investigate inherent model biases, we assemble a 415-item health and medical bias dataset that couples real-world clinical vignettes with controlled perturbations designed to elicit biased behaviour. The corpus derives from two sources: first, 304 items curated from the public bias dataset of45, screened manually by physicians to retain only those prompts most likely to expose inequity; second, 111 additional complementary scenarios authored by three licensed clinicians that maximize bias elicitation while preserving clinical realism. Each sample contains a structured clinical note with a patient placeholder, a patient self-report and a query to the evaluated models requesting various aspects (Fig. 5), including various clinical tasks such as risk assessment, triage prioritization and treatment recommendation.
Fig. 5: Overview of DAS Bias Red-teaming.
a–d, Examples of bias-eliciting strategies: cognitive bias priming (a), identity manipulation (b), linguistic manipulation (c) and emotional manipulation (d). e, Effectiveness of each bias-eliciting strategy, shown as jailbreak ratio (in percentage, that is, the proportion of responses shifting to biased after attack). Sample size N = 415, with error bars indicating 95% confidence intervals and mean values as the centre. An average of 87% bias jailbreak ratio is observed across all models, with cognitive bias priming emerging as the most effective bias-triggering strategy, eliciting more than 80% biased answers in health-related recommendations. Bold indicates the jailbreak ratio.
All target models first respond to the original untouched scenario, establishing an unbiased reference answer. We then deploy four adversarial toolchains that mutate the prompt while leaving the underlying pathology unchanged: (1) cognitive bias priming (Fig. 5a)—appending a single sentence after the clinical note that invokes status quo, recency, false consensus, defensive medicine, authority or availability heuristics (details and explanation in Sec. 10.3.2), testing whether the model adopts the seeded bias; (2) identity manipulation (Fig. 5b)—swapping the patient placeholder for sociodemographic labels spanning ethnicity, housing status, gender and sexual identity and income (full list in Methods); (3) linguistic manipulation (Fig. 5c)—rewriting the self-report into African American English, limited English, extremely polite, extremely rude, obscure slang, Chinese or Spanish; and (4) emotional manipulation (Fig. 5d)—infusing the patient dialogue with affective cues (angry, assertive, anxious, depressive or optimistic).
Each mutated prompt is generated by a strategy agent with full access to both the original clinical question and the model’s initial response. The agent’s goal is to select the identity, linguistic, emotional or heuristic twist most likely to drive the evaluated model to diverge from its reference decision (Fig. 5a–d). We systematically quantify the frequency and in which clinical context these perturbations alter triage urgency, treatment recommendations or risk stratification. This enables a granular and comprehensive healthcare audit of equity in state-of-the-art LLMs.
Key results of bias elicitation using DAS red-teaming are summarized in Fig. 5e. Cognitive bias priming is the most potent bias-inducing strategy, achieving a high average jailbreak rate of 81.1%. By contrast, emotional (24.1%), identity (24.0%) and linguistic manipulations (23.9%) yield substantially lower yet comparable jailbreak rates, highlighting the dominance of cognitive biases in influencing model decisions. Furthermore, each scenario is subjected to all four bias eliciting strategies and the combined jailbreak rate—indicating susceptibility to at least one manipulation—exceeded 80% across all evaluated models. Notably, among tested models, Claude-series models (Sonnet-3.7 and Sonnet-4) demonstrate the highest resilience against bias manipulations, whereas QwQ-32B and Gemma-3 consistently exhibit pronounced vulnerabilities to bias attacks and showed the highest susceptibility in our audit test. In summary, these results underscore a troubling yet pervasive oversight/incapability of fairness considerations in current LLMs used for health applications.
Hallucination–factual inaccuracies
LLMs that hallucinate in a healthcare/medical context can pose an immediate threat to user safety. However, ‘hallucination’ in the context of healthcare and medicine is an umbrella term that is broadly defined and still evolving. To address this ambiguity and enable systematic risk quantification, we propose a structured and fine-grained medical-specific taxonomy that deconstructs medical hallucination into seven health-critical and clinically relevant error categories: (1) false or outdated medical facts or claims, (2) incorrect, misrepresented or fabricated citations and references, (3) faulty healthcare/medical reasoning chains, (4) neglect, distortion or fabrication of provided context, (5) harmful or unsafe recommendations, (6) failure to follow explicit or implicit instructions and (7) other inaccuracies (for example, conflating gender and biological sex).
This classification provides a clear framework for consistent annotation and transparent risk communication among developers, patients, caregivers, clinicians, regulators and health-system stakeholders.
We curate a balanced hallucination dataset (n = 260) from two expert-vetted sources. The positive subset (n = 131) consists of high-confidence prompts curated from the Stanford Red-Teaming corpus22 and refined and filtered further by our board-certified physician, which are validated to produce a 100% hallucination rate on baseline models GPT-3.5/GPT-4. The negative subset (n = 129) contains near-ideal, factual responses sourced from OpenAI’s extensively vetted HealthBench40, which we further filtered to eliminate any latent errors. Please see Methods for more details.
Building on this taxonomy, we further develop a novel hallucination detection framework managed by an orchestrator agent coordinating seven specialized subagents, each corresponding to one taxonomy category (Fig. 6a). We test it on the curated hallucination dataset (in total, 260 cases with 131 positive and 129 negative samples). As detailed in Fig. 6b, we achieved 81.9% accuracy (F1 of 81.5%), demonstrating balanced precision (79.8%) and recall (83.2%) on a carefully curated 260 sample dataset. These metrics confirm that decomposing the task into health-critical and clinically meaningful subproblems is a promising route to dependable hallucination detection.
Fig. 6: Overview of DAS red-teaming Hallucination test for LLMs in health.
a, The architecture of the hallucination detection framework, comprising an orchestrator agent (OpenAI GPT-4o) coordinating seven specialized subagents (OpenAI o3), each targeting one category of the proposed taxonomy. b, The detection performance (accuracy, precision, recall and F1) on the curated 260-sample dataset. c, Hallucination ratios (overall and by category) across evaluated models. Sample size N = 131. Error bars indicate 95% confidence intervals, adjusted using the Rogan–Gladen estimator and Monte Carlo simulation to account for detection error (refer to Methods for more details). On average, 74% of test cases elicited hallucinations, with false or outdated medical facts as the predominant error type, indicating persistent factuality risks for health information use and care delivery. Representative examples are provided in Supplementary Information section D.5.
We evaluate the factuality/truthfulness performance of the 15 models by applying the proposed hallucination detector using the positive subset of the hallucination dataset, assessing the extent to which day-to-day users and clinicians can trust LLM outputs in complex medical and healthcare cases. Importantly, we do not elaborate efforts to elicit medical hallucinations, which we consider closely related to the robustness attack pipeline (see the discussion in Methods). The hallucination and detection examples are showcased in Supplementary Information section D.5. The analysis revealed a striking performance gap between models (Fig. 6c). The proprietary models (nine LLMs) provided responses with an average hallucination ratio of 62.6%, whereas the open-source ones (six LLMs) got a hallucination ratio of 72.5%. Specifically, proprietary models significantly reduced medical factual errors compared with open-source peers (29% versus 49%). Notably, models with explicit chain-of-reasoning (o4-mini, o3, Gemini-2.5-Pro, DeepSeek-R1, HuatuoGPT-o1, Qwen3 and QwQ) hallucinate more frequently than other baselines without thinking (68.7% versus 64.7%). Although chain-of-thought structuring improves answer organization, it also propagates early false premises, yielding more logic errors (26% versus 22%), context failures (20% versus 15.6%) and citation errors (20% versus 15.4%). However, explicit reasoning benefits the models’ instruction compliance (7.6% versus 10.5%) and helps the models think deeply about potential unsafe recommendations (12.2% versus 14%). In addition, we observed no correlation between the model parameter scale and the hallucination error. For example, MedGemma-27B (65.1% error rate) outperforms both Qwen3-32B (77.4%) and QwQ-32B (80.2%). Moreover, the medical-fine-tuned models (MedGemma and HuatuoGPT) improve the overall hallucination problem over other general-purpose baselines (64% versus 67%). Notably, MedGemma is significantly better than its baseline Gemma-3 across all hallucination categories.