Our search identified 92 studies meeting the inclusion criteria (Fig. 2). These studies reveal substantial diversity in medical biases across clinical scenarios and specialties. Table 1 presents an overview of this scoping review. In the following sections, we synthesize existing methodologies for detecting and mitigating bias, identify critical research gaps, and discuss potential directions for future solutions.
Table 1 A taxonomy of medical biases along with their corresponding detection and mitigation approachesBias in large AI models for medicine and healthcare
Medical bias in large AI models can originate from multiple stages of the model development pipeline and manifest in diverse forms across clinical settings and disease domains.
Regarding the sources of bias, we conceptualize medical bias as arising across three interrelated levels: data-level, model-level, and evaluation-level bias31,38. Data-level bias may result from limited dataset size and diversity, poor data quality, lack of representativeness, or subjective annotation31. Model-level bias may emerge from architectural design choices, loss function design, or post-training alignment procedures31. Evaluation-level bias may be introduced through optimization objectives, benchmark selection, demographic categorizations, or evaluation metric selection31. Figure 3 presents a conceptual framework of large AI model applications in medicine and healthcare, highlighting potential sources of medical bias throughout the process.
Fig. 3
Conceptual framework of potential sources of medical bias in large AI models.
To characterize how medical bias manifests in practice and to distinguish it from prior general AI fairness research, we develop a dual taxonomy that organizes the literature along two principal axes: (1) medical scenarios, such as clinical decision support and medical education; and (2) clinical specialties, such as cardiology and pulmonology. This taxonomy enables a more nuanced analysis of medical bias from both functional and clinical perspectives, illustrating the broad range of contexts in which bias may emerge. By structuring the literature in this manner, we facilitate the identification of high-risk scenarios and specialties and provide a foundation for more targeted evaluation and mitigation strategies. Given the rapid evolution of large AI models and their applications in healthcare, this taxonomy should be understood as an organizing and illustrative framework derived from the included studies, rather than an exhaustive classification of all possible forms of medical bias. Additionally, we analyze the density and distribution of current research on medical bias in large AI models and discuss the potential causes of this research landscape in the “Research density and distribution.”
Bias across medical scenarios
To elucidate how medical bias arises in practical applications of large AI models, we introduce a scenario-based categorization that mirrors typical use cases in real-world medicine and healthcare, including clinical decision support bias, patient communication bias, medical documentation bias, and medical education bias. In Table 2, each type of medical bias is illustrated with representative examples and explanations of its manifestations in AI models.
Table 2 Examples of medical bias across scenarios
Clinical decision support bias
Clinical decision support bias arises when large AI models that assist clinicians in diagnostic reasoning or treatment planning produce systematic disparities in diagnostic performance or recommended actions across demographic groups under equivalent clinical conditions. Such bias may lead to inappropriate or unequal clinical decisions. For example, Omar et al.52 evaluated 1000 emergency department cases across 31 sociodemographic variations and nine LLMs, resulting in 1,728,000 total model runs. Cases labeled as Black and unhoused were more likely to be judged as requiring mental health assessment than cases labeled as unhoused alone or as White and unhoused (79.8% vs. 73.5% and 76.5%, respectively). These cases also received higher invasiveness scores than the control group (1.18 vs. 1.12; P < 0.001).
Patient communication bias
Patient communication typically involves conversational agents or chatbots powered by large AI models that interact with patients in natural language to answer health-related questions, explain medical terms, and offer personalized health counseling. These systems support the continuous process of monitoring, maintaining, and improving an individual’s health through prevention, early detection, lifestyle guidance, and chronic disease management. Patient communication bias occurs when such models generate information or recommendations whose clinical appropriateness varies across demographic groups or individuals23,37,53,54,55,56. For example, Nastasi et al.23 evaluated 96 patient advice-seeking vignettes and found that prompts describing patients without insurance were significantly more likely to receive cost- or access-tailored responses (OR 9.76, 95% CI: 3.79–28.1; P < 0.001). In one high-risk chest pain vignette, the uninsured patient was directed either to a community health center or to the emergency department, whereas the insured counterpart was directed exclusively to the emergency department.
Medical documentation bias
Medical documentation bias occurs in large AI models that extract, summarize, interpret, or generate clinical notes and medical reports. It refers to disparities in the tone, completeness, or accuracy of the generated documentation across demographic groups or clinical contexts26,57. Such bias often stems from domain shifts or imbalanced textual corpora, leading to omissions of key details or the use of stereotypical language, which may in turn affect clinicians’ interpretation and decision-making. For example, Yang et al.57 analyzed 36,800 generated medical report responses and identified race-associated differences in GPT-3.5-generated documentation. Among 200 generated patient reports, 16 contained biased rephrasings of patient information, including unsupported race-linked details, such as fabricating a trip to South Africa for a Black male patient.
Medical education bias
Medical education bias occurs in large AI models that produce understandable medical explanations, generate personalized educational materials, or provide diagnostic and surgical simulations for medical professionals. It arises when these models or their synthetic datasets disproportionately represent certain demographic groups or embed stereotypical assumptions56. Such imbalances can distort both downstream models and medical education materials, misrepresenting the real-world diversity of diseases and populations. For example, Zack et al.56 found that when GPT-4 generated sarcoidosis vignettes for potential use in medical education, 966 of 1000 cases (97%) described a Black patient, and 810 cases (81%) described a Black female patient. Such synthetic cases may overrepresent epidemiological associations and reinforce racialized disease stereotypes in medical education materials.
Bias across clinical specialties
Complementing the scenario-based taxonomy, we further categorize existing studies by clinical specialty to examine how medical bias manifests across disease domains. This perspective is important because bias patterns and risks are often disease-specific. We identify 34 studies focusing on particular conditions and group them into major specialties, such as Cardiology, Pulmonology, and Ophthalmology. This specialty-based view highlights both shared and domain-specific bias challenges, offering a clearer basis for developing targeted evaluation and mitigation strategies.
Medical bias in psychiatry
Chansiri et al.20 found that GPT-3.5 associated women with borderline personality disorder 8.6 times more often than with narcissistic personality disorder and associated men with narcissistic personality disorder 4.3 times more often than with borderline personality disorder. In emergency mental health decision-making, Omar et al.52 reported that patient descriptors indicating LGBTQIA+, transgender, or unhoused status were associated with higher rates of recommendations for mental health assessment.
Medical bias in cardiology
GPT-4 was reported to change its diagnosis when only patient sex was changed from Black male to Black female, a discrepancy that was not observed among clinicians35. A more recent cardiovascular-specific study further found that, in sex-swapped real patient notes, GPT-4o showed significant differences in diagnostic accuracy in 11 of 50 cases, with lower accuracy for female patients than for male patients in 8 cases58.
Medical bias in dermatology
In dermatology, medical bias often manifests as disparities related to skin tone, sex, and age in image generation, diagnosis, and treatment recommendations. Joerg et al.59 found that, among 4000 AI-generated dermatologic images, 89.8% depicted light skin and only 10.2% depicted dark skin; ChatGPT-4o, Midjourney, and Stable Diffusion significantly underrepresented dark skin. In clinical vignettes, Kim et al.35 reported that AI chatbots were more likely to recommend isotretinoin for male patients with acne than for female or transgender patients.
Medical bias in pulmonology
Glocker et al.60 reported that a chest radiography foundation model showed reduced performance for female patients in no-finding classification and for Black patients in pleural effusion detection, with subgroup performance reductions of 6.8–7.8% and 10.7–11.6%, respectively. Similarly, Yang et al.22 showed that strong overall performance of expert-level medical imaging foundation models can mask substantial subgroup gaps across age, sex, and race. These findings indicate that high aggregate accuracy is insufficient for evaluating fairness in pulmonary imaging tasks.
Medical bias in oncology
Gourabathina et al.61 evaluated GPT-4, Llama-3-70B, Llama-3-8B, and Palmyra-Med. The study introduced non-clinical perturbations to patient messages, including changes in sex, uncertainty, expressive language, typographical errors, whitespace, and capitalization, and assessed whether model treatment recommendations changed. These perturbations produced an average treatment-recommendation shift of 7.31% and an average reduced-care error rate of 4.12%.
Medical bias in ophthalmology
Luo et al.42 introduced a fairness-oriented ophthalmology benchmark and showed that FairCLIP improved both performance and fairness, reducing the race-related demographic parity difference from 15.35 to 6.07% and the sex-related demographic parity difference from 4.34 to 0.84% in a zero-shot setting. Restrepo62 further reported substantial language gaps in ophthalmological question answering, with GPT-4 achieving 63.4% accuracy in English compared with 51.8% in Filipino and 50.6% in Hindi.
Medical bias in emergency medicine
Omar et al.52 analyzed more than 1.7 million outputs from 9 LLMs across 1000 emergency department cases and found that LGBTQIA+, transgender, and unhoused labels were associated with more frequent mental health assessment recommendations, whereas high-income cases were more likely, and low-income cases less likely, to receive advanced testing recommendations. Nastasi et al.23 also reported that insurance status affected ChatGPT responses in a high-risk chest pain vignette, with an uninsured patient sometimes directed to a community health center rather than to the emergency department.
Medical bias in pain medicine
Young et al.63 evaluated opioid recommendations across race and sex variations and found that GPT-4 recommended opioids in 82.9% of cases and Gemini in 87.3%, although race and sex did not significantly affect the measured recommendation outcomes in that study. In contrast, Deb and Rodman64 found that false beliefs about race were more common in Gemini Pro than in GPT-4 and were associated with lower odds of higher pain ratings for Black patients when such beliefs were present.
Medical bias in infectious disease
Sallam et al.65 reported lower response quality for Arabic than English infectious disease queries, with mean CLEAR scores of 4.1 and 4.6, respectively. Criss et al.66 found that ChatGPT-4.0 generated 18 discrimination-related and 47 stigma-related references in HIV advice, with response patterns varying by race, sexual orientation, and sex identity.
Medical bias in neurology
In epilepsy diagnosis and treatment, Bhansali et al.67 reported a large socioeconomic gap in model diagnostic accuracy, with high-socioeconomic-status cases achieving 58% accuracy compared with 13% for low-socioeconomic-status cases before bias-mitigation prompting.
Across clinical domains, race, sex, and skin tone consistently emerge as the dominant axes of bias in large AI model performance. However, the manifestations and mechanisms of these biases vary by specialty, ranging from diagnostic disparities in cardiology to representational imbalance in dermatology and data-driven prevalence distortion in oncology. These patterns indicate that generic evaluations are insufficient; fairness requires domain-aware auditing and specialty-specific mitigation strategies that account for each field’s data characteristics and clinical workflows. Advancing fairness in medical AI thus demands aligning technical assessment with the practical and ethical realities of individual specialties.
Medical bias detection
Bias detection, also known as bias testing or fairness testing, aims to identify and quantify potential biases and unfairness in AI models41,68. Existing studies employ two major types of criteria for bias detection: answer consistency and statistical measurements. Answer consistency is particularly useful for detecting bias in single cases or small sets of clinical cases, because it identifies bias by comparing model responses before and after demographic perturbation. Statistical measurements are particularly useful in settings with well-established datasets that contain explicit sensitive attributes for each data instance. They compare model outputs, accuracy, recommendation rates, error rates, or linguistic features across demographic subgroups, often supported by statistical significance testing. These two major bias detection methods are reliable, widely validated, and complementary. In addition, in some complex healthcare scenarios in which answer consistency and statistical assessment may be unavailable, manual bias assessments based on human experts or large AI models provide a flexible alternative. In the following sections, we introduce these types of bias assessment criteria in detail.
Input generation
In the context of large AI models for medicine and healthcare, it is often infeasible to find patient pairs that are identical in all respects except for the considered sensitive attribute. As a result, medical bias detection typically involves constructing counterfactual variants of real clinical vignettes to evaluate the bias in the output of large AI models24. This approach aligns with established software testing practices that deliberately modify input data to trigger abnormal system behavior, thereby assessing the reliability, robustness, and correctness of the system69,70. These strategies can be broadly categorized into synthetic or simulated patient generation and mutation-based variations of existing cases.
Synthetic or simulated patient generation
Yeo et al.37 utilized simulated patients derived from digital standardized patients to assess biases in GPT-4’s provision of mental health support. Zack et al.56 evaluated GPT-4-generated clinical vignettes as a form of simulated patient generation for medical education and clinical reasoning. They generated patient presentations across multiple medical conditions and compared the demographic distributions of these cases with real-world prevalence estimates. However, the synthetic cases may not fully capture real-world clinical complexity, including comorbidities, longitudinal disease trajectories, documentation noise, clinician–patient interactions, and evolving clinical workflows. Their validity also depends on whether the generated demographic variations remain clinically plausible and whether the underlying assumptions about relationships between sensitive attributes and health outcomes are appropriate. Moreover, vignette-based evaluations may miss subtle or implicit biases that emerge only in richer clinical contexts, multi-turn interactions, or real-world deployment settings.
Mutation-based variations
Many studies mutate existing prompts or cases by systematically altering sensitive attributes such as sex and race. Several studies employed manual or structured attribute variations. Zack et al.56 adapted 19 expert-generated medical education cases from the NEJM Healer platform by varying patient sex (male or female) and race (Asian, Black, White, or Hispanic). Kim et al.35 constructed 19 clinical vignettes spanning multiple specialties, systematically varying sex, race, and socioeconomic status while ensuring these attributes did not alter the standard of care. They found that GPT-4 and Bard exhibited notable biases in treatment recommendations, particularly for women, Hispanic patients, and transgender individuals. Ito et al.71 used 45 standardized clinical vignettes, each with a correct diagnosis and triage level, and assigned one of four racial/ethnic identities (Black, White, Asian, Hispanic). GPT-4’s diagnostic and triage performance was comparable to board-certified physicians and showed no significant variation across races. Benkirane et al.72 created counterfactual clinical scenarios by filtering out cases related to pregnancy or women’s health, removing explicit ethnicities, and applying specialty and publication year filters. They then generated male, female, and sex-neutral versions along with ethnic variations (Arab, Asian, Black, Hispanic, White). Pfohl et al.24 developed two counterfactual datasets—CC-Manual and CC-LLM—to evaluate bias in Med-PaLM and Med-PaLM 2. Physicians reported bias in 13% of CC-Manual pairs, while health equity experts reported 18%. For CC-LLM, physicians observed a lower bias rate, whereas experts reported similar levels across both datasets. Automated adversarial approaches have also been explored. Poulain et al.25 applied red-teaming strategies, mutating questions via adversarial prompting and rotating through different patient demographics.
However, mutation-based detection also has important limitations. Altered cases may not always remain clinically plausible, and one-attribute-at-a-time mutations may fail to capture intersectional bias involving combinations of race, sex, age, socioeconomic status, language, disability, or other social factors. In addition, automated adversarial mutations may introduce unrealistic or overly salient cues that do not reflect authentic clinical scenarios. Consequently, findings from mutation-based detection should be interpreted alongside clinical plausibility assessment and complemented by real-world validation.
Mutation-based approaches can support intersectional analysis when multiple clinically plausible attributes are varied simultaneously, but more systematic methods are needed to ensure clinical equivalence.
Bias evaluation
Bias detection by answer consistency checking
Many studies assess medical AI bias by measuring answer consistency across counterfactual or demographically varied inputs, where only sensitive attributes (e.g., race, sex, socioeconomic status) differ. Pfohl et al.24 used two counterfactual datasets (CC-Manual and CC-LLM) to compare bias rates in Med-PaLM models as judged by physicians and health equity experts. Xiao et al.73 constructed three sets of counterfactual pairs (White vs. Black, Male vs. Female, and High Income vs. Low Income) from 801 USMLE-style clinical vignettes and revealed significant biases related to race, sex, and socioeconomic status in five influential LLMs, including GPT-4.1 and Claude-3.7-Sonnet. Zack et al.56, Kim et al.35, and Ito et al.71 systematically varied sensitive attributes in clinical vignettes or prompts while holding all other information constant, then compared outputs for differences in diagnoses, recommendations, sentiment, or word choice, often applying statistical tests (e.g., Mann-Whitney, chi-squared).
Bias detection with classic fairness metrics
Naturally, many classic fairness metrics can be adopted to measure the bias in large AI models for health, such as Demographic Parity, Equal Opportunity, and Equalized Odds.
Jin et al.74 applied Demographic Parity, Equal Opportunity, and Equalized Odds in their FairMedFM benchmark for medical imaging. Luo et al.42 introduced Harvard-FairVLMed, a vision-language medical dataset with demographic attributes, ground-truth labels, and clinical notes, enabling fairness analysis in large VLMs; they reported results using Demographic Parity, Equalized Odds, and area under the curve (AUC) differences. Fayyaz et al.75 employed Demographic Parity and Equal Opportunity to measure fairness in their evaluations. Benkirane et al.72 used Equalized Odds alongside an accuracy consistency measure, and also proposed the SkewSize metric to capture the distribution of bias-related effect sizes across classes.
Bias detection with AI metrics
Beyond classical fairness measures, many studies quantify bias in large healthcare AI models using task-specific performance metrics. Yang et al.22 measured underdiagnosis disparities in chest X-ray interpretation by comparing false-negative rates and false-positive rates across race, sex, and age groups. Jin et al.74 evaluated disparities in accuracy, AUC, and Dice similarity coefficient within a medical imaging benchmark. Alongside Poulain et al.25, they also compared predictive probability distributions across demographic groups and applied statistical tests to assess significance. Benkirane et al.72 quantified bias in LLM-driven clinical decision-making using accuracy consistency and SHAP value analysis. Goh et al.76 compared clinical decision accuracy between White men and Black women in chest pain evaluation scenarios. Luo et al.42 evaluated fairness in glaucoma diagnosis using a vision-language medical dataset by reporting AUC differences across demographic groups.
Bias detection with domain-specific metrics
Yeo et al.37 used the Linguistic Inquiry and Word Count (LIWC-22) tool, which is text-analysis software that quantifies psychological and linguistic components present in spoken or written speech. Specifically, LIWC-22 calculates the percentage of words falling into psychologically motivated categories and reports four standardized summary measures, namely Analytical Thinking, Clout, Authenticity, and Emotional Tone, which can be compared across groups for bias evaluation.
Bias detection with an LLM as a judge
Swami Nathan et al.77 argued that to deploy LLMs within health systems at scale, automated bias evaluations are needed. To this end, they studied the performance of LLM agents in judging the bias in LLM responses to race-based medicine questions and reported the percentage of LLM responses that did not contain debunked race-based content. However, the use of LLMs as judges also introduces important limitations. Because LLM-based evaluators may themselves exhibit bias or generate hallucinated judgments, their robustness and trustworthiness cannot be assumed a priori. In addition, their evaluations may vary across prompts, model versions, and evaluation settings, raising concerns about consistency and reproducibility.
Bias detection by human expert assessment
While answer consistency and group-wise statistics can support automatic bias detection, they may be insufficient in certain cases. For instance, when sensitive attributes are implicit, constructing counterfactual pairs becomes challenging, making answer consistency inapplicable. Additionally, group-wise statistics are ineffective for detecting bias in a single AI-generated response. In such situations, structured human review, supported by clear rubrics and conducted by diverse rater groups, can help uncover fairness issues. Human expert evaluations provide clinically grounded judgments, but their reproducibility may vary across evaluators, clinical specialties, levels of experience, and scoring rubrics. Reproducibility can be improved through the use of predefined evaluation criteria, multiple independent reviewers, calibration exercises, and the reporting of inter-rater agreement when feasible. However, many existing studies do not report these methodological details consistently, which limits reproducibility and cross-study comparability.
Pfohl et al.24 evaluated Med-PaLM 2 with three rater groups (physicians, health equity experts, and consumers) and three simple rubrics: (i) independent ratings of single answers, (ii) pairwise comparisons to a reference answer, and (iii) counterfactual review of paired questions that differ only in identity cues. They also reported inter-rater reliability (Randolph’s κ, Krippendorff’s α) and showed that results depend on how ratings are aggregated (any-vote vs. majority). On counterfactual pairs, experts flagged bias more often than physicians; consumer raters flagged bias more often than either group on a mixed question set, underscoring the value of multiple perspectives.
Nastasi et al.23 had two physicians review ChatGPT (GPT-3.5) on 96 patient-style vignettes. Most answers matched guidelines (97%), but advice sometimes changed with insurance status (e.g., suggesting a community clinic instead of the emergency department for an otherwise identical high-risk chest-pain case).
Omiye et al.78 tested four commercial models (GPT-3.5, GPT-4, Bard, Claude) on nine questions about debunked race-based practices, running each question five times. Two physicians rated each response, with a third adjudicating ties. All models sometimes reproduced race-based medicine (e.g., incorrect estimated glomerular filtration rate or lung-capacity formulas), and answers varied across runs—highlighting the need for repeated queries and expert adjudication.
Medical bias mitigation
In the “Medical bias detection,” we reviewed the existing literature on medical bias detection in large AI models. In this section, we introduce the corresponding approaches to mitigating medical bias. According to previous bias and fairness research, bias mitigation strategies are typically categorized into pre-processing, in-processing, and post-processing methods, based on whether the mitigation occurs before, during, or after model training. Table 3 presents the descriptions of the three categories of bias mitigation strategies along with examples.
Table 3 Taxonomy of bias mitigation strategies for large AI modelsPre-processing
Pre-processing strategies for mitigating medical bias in large AI models focus on modifying or enhancing training data prior to model training or fine-tuning. These data-level interventions aim to reduce bias by addressing imbalances or stereotypes embedded in the input data31. Common techniques include data augmentation79, data filtering80, instance reweighting81, and synthetic data generation82,83,84.
In-processing
In-processing strategies aim to mitigate bias by modifying model internals during training or fine-tuning31. These methods introduce fairness-aware interventions into the model architecture, objective functions, or parameter update procedures. While theoretically powerful, their practical adoption in large AI models is limited by resource constraints, architectural complexity, and restricted access to model internals, particularly for closed-source models. We categorize representative in-processing methods as follows.
In-processing approaches
Model fine-tuning. Fine-tuning is a typical in-processing approach that involves partially retraining model parameters using additional data under different training settings31,85.
Architecture modification. Architecture-level modifications alter core model components, such as the number and configuration of layers or attention heads31,86. Although explored in smaller models, such interventions are rarely applied to full-scale large AI models due to retraining cost and implementation complexity.
Loss function modification. Fairness objectives can be integrated into the loss function via regularization terms, auxiliary tasks, or adversarial constraints31. For example, contrastive and reinforcement learning have been used to steer models away from sensitive-attribute-based reasoning, promoting more equitable outputs in clinical scenarios87,88.
Decoding strategy modification. Bias can be mitigated during text generation by constraining decoding procedures. For instance, fairness-aware beam search or sampling penalties can suppress biased completions and promote demographically neutral outputs, without altering model weights89,90.
Practice of in-processing approaches
Luo et al.42 proposed FairCLIP, a framework designed to improve fairness during the pre-training phase. FairCLIP minimizes the Sinkhorn distance between the overall sample distribution and the distributions corresponding to each demographic group. It has been shown to significantly outperform CLIP91 in terms of both performance and fairness.
Jin et al.74 checked various in-processing unfairness mitigation methods for traditional neural networks on large models for medical imaging. They found that existing unfairness mitigation strategies are not consistently effective and often result in poor fairness-utility trade-offs, sometimes even degrading both fairness and overall performance.
Hasheminasab et al.92 demonstrated that fine-tuning large AI models on local datasets significantly reduces bias and enhances performance in specific healthcare contexts, with improvements of 16–27% in F1 scores and 21–31% in precision. However, this approach may limit global generalization capabilities. Similarly, Yan et al.93 adopted a two-phase fine-tuning approach (frozen then unfrozen backbone) across ophthalmology, radiology, and dermatology domains. Their results showed significant gains in both fairness and performance, with sex integration improving fairness by 2.5% and performance by 8.6% in ophthalmology. Zahraei and Shakeri94 fine-tuned ChatDoctor using PEFT/LoRA techniques95 on two custom datasets, creating the EthiClinician model, which significantly outperformed existing models, including GPT-4, in both bias mitigation and diagnostic accuracy.
Zheng et al.96 introduced an adversarial debiasing framework based on variational auto-encoders (VAEs), wherein 3D CT embeddings are mapped to a latent space optimized to remove sensitive demographic information (e.g., age, sex) via an adversarial network—while preserving downstream predictive features—and the method is evaluated on NLST data using metrics such as attribute prediction accuracy, cancer risk prediction, equal opportunity difference, and robustness to data poisoning.
Research on the ISIC dataset by Munia and Imran97 introduced DermDiT, which integrates the DiT architecture with LLaVA-Med textual guidance. The model achieves the lowest FID and highest MS-SSIM scores, demonstrating both high fidelity and enhanced diversity in synthetic dermoscopic images. Classification models trained on these synthetic datasets outperform those trained on real ISIC images in terms of recall and F1-score, particularly for minority subgroups, thereby mitigating diagnostic bias without compromising overall performance.
Some studies report more nuanced or limited outcomes. Khan et al.98 applied fine-tuning on balanced datasets using the AdamW optimizer for chest X-ray diagnosis across six large AI models. While this approach reduced demographic biases, biases toward majority groups persisted, and female patients consistently experienced lower performance.
Jin et al.74 observed similar limitations in medical imaging tasks involving chest X-rays, skin lesions, and eye conditions. Fine-tuning strategies often improved fairness metrics but at the cost of overall performance, suggesting that existing approaches require significant adaptation to effectively address bias in large AI models. Hastings99 highlighted that fine-tuning through data augmentation and reinforcement learning offers limited effectiveness in reducing bias, particularly for closed models like GPT-4, underscoring the need for complementary mitigation strategies.
Several studies emphasized mixed results depending on the bias dimension and demographic group. Benkirane et al.72 employed fine-tuning on GPT-4o mini with a balanced dataset. While sex bias was mitigated successfully, ethnic bias reduction showed inconsistent outcomes, with improvements for some ethnicities but the introduction of new disparities for others. Poulain et al.25 found that fine-tuning improved overall model performance but failed to significantly reduce bias, and in some cases, introduced concerning disparities, indicating that domain-specific training alone is insufficient for ensuring fairness. Zack et al.56 reported mixed results using various fine-tuning approaches, including LoRA and domain adaptation. These methods were effective in reducing specific biases but sometimes compromised overall performance or failed to generalize across populations.
Post-processing
Given the large parameter scale and limited transparency of most large AI models, even those released as open source, their internal training data, optimization procedures, and architectural details remain largely opaque to users. This opacity poses significant challenges for implementing pre-processing or in-processing mitigation techniques, especially when fine-tuning is computationally infeasible or API-only access constrains intervention. As a result, post-processing emerges as a practical and model-agnostic strategy for mitigating bias by modifying the model’s outputs after generation, without altering its underlying parameters31,41.
Post-processing approaches
Output rewriting. This approach identifies biased, stereotypical, or otherwise harmful elements in the generated text and revises them using either rule-based substitution or neural re-ranking techniques. Unlike decoding-time filtering, output rewriting operates on the fully generated response and aims to preserve the semantic intent while improving fairness or inclusiveness. These methods are particularly applicable in domains such as clinical report summarization or recommendation generation, where surface-level stereotypes can introduce patient harm.
Output ensembling. Inspired by fairness methods in classical machine learning, output ensemble strategies generate multiple completions for a given prompt and apply aggregation techniques—such as majority voting, score-based selection, or diversity-aware re-ranking—to synthesize a fairer final response82. This technique leverages the stochasticity of decoding in large AI models to introduce output variation and reduce systemic bias.
Prompt engineering. Prompt engineering refers to the deliberate design and structuring of input prompts to steer large AI models toward desired behaviors, outputs, or ethical constraints100. By modifying the wording, context, or format of the prompt, such as explicitly specifying fairness objectives or embedding counter-stereotypical examples, researchers can mitigate biased responses without altering the model’s underlying parameters. Zhang et al.101 found that asking GPT-3.5 to explain its reasoning prior to providing an answer is able to mitigate sex and racial biases in clinical management of acute coronary syndrome.
Practice of post-processing approaches
In medical question answering, inputs or prompts can be modified to instruct the model to avoid generating biased answers based on sensitive attributes. By prepending additional static or trainable tokens to an input, prompt engineering conditions the model’s output generation in a controllable manner, helping to mitigate bias and guide the model toward more fair and accurate responses31.
Common techniques include zero-shot prompting (task description only), few-shot prompting (task examples provided), and chain-of-thought (CoT) prompting (step-by-step reasoning). Despite its flexibility, prompt engineering faces challenges such as ambiguity, sensitivity to input variations, and the need for iterative experimentation. This section reviews recent studies that explore the use of prompt engineering for medical bias mitigation, categorized by the specific strategies employed and their reported effectiveness.
Chain-of-thought prompting. CoT prompting, which encourages models to reason step-by-step, has shown significant promise in reducing biases in clinical decision-making tasks. Poulain et al.25 demonstrated that CoT prompting outperformed zero-shot and few-shot approaches in pain management scenarios, effectively eliminating demographic-based disparities in treatment recommendations. Similarly, Ke et al.102 showed that CoT prompting, combined with bias-aware strategies, enabled medical large AI models to incorporate systematic reasoning and diverse perspectives. However, the authors highlight that the effectiveness of CoT prompting can vary depending on the medical domain and the type of bias being addressed.
Few-shot prompting. Few-shot prompting, where examples are provided to guide the model, has been widely tested for bias evaluation and mitigation. Zahraei and Shakeri94 employed few-shot prompting to generate ambiguous test cases for evaluating demographic biases in medical scenarios. While effective in producing controlled test cases, the study did not conclusively demonstrate its role in bias reduction. Schmidgall et al.103 compared few-shot demonstrations with other strategies (bias education and one-shot prompting) on diagnostic tasks. The results indicated that while all methods reduced cognitive biases to some extent, none completely eliminated their impact, with GPT-4 showing the greatest improvement.
Fairness-aware prompting. Fairness-aware prompting strategies explicitly aim to incorporate fairness constraints into the model’s outputs. Wang et al.104 proposed fairness calibration prompting for mental health tasks, including stress prediction and wellness assessment. Their results showed that fairness-aware prompting significantly reduced demographic biases while maintaining high task performance, demonstrating its applicability in sensitive medical contexts.
Several studies tested the efficacy of explicitly instructing models to avoid biased responses. Zack et al.56 noted that while such instructions can reduce demographic biases in GPT-4’s medical reasoning, this approach may lack practicality in real-world clinical settings due to the effort required for explicit prompt design. Hastings99 further argued that simple bias-avoidance instructions are insufficient because language models lack self-awareness and consistent world models. They recommend combining prompt engineering with external verification methods and demographic controls for better results.
Model-specific variability in prompt effectiveness. Benkirane et al.72 highlighted the variability in prompt engineering effectiveness across models and demographic categories. While some prompts successfully mitigated sex bias, the results were inconsistent for ethnic biases, sometimes introducing new disparities. These findings underscore the need for model-specific debiasing approaches and careful prompt optimization tailored to the target demographic groups.
The studies collectively demonstrate that prompt engineering strategies, such as CoT reasoning, fairness-aware prompting, and few-shot demonstrations, can help mitigate biases in large medical models. However, the results vary across medical domains, demographic categories, and model architectures. While CoT prompting and fairness calibration approaches show promising results in clinical decision-making and mental health applications, explicit bias-avoidance instructions and simple prompts often fail to address the root causes of bias. Furthermore, no single strategy has completely eliminated bias, highlighting the need for complementary approaches, such as integrating external fairness constraints or post-hoc validation mechanisms.
Because pre-processing and in-processing methods often require more resources for model retraining or fine-tuning, post-processing methods, such as prompt engineering, have been more widely explored and validated in the era of large AI models.
Research density and distribution
Medical bias in large AI models has attracted growing attention from the academic community, with related research increasing significantly since 2023. We analyze the distribution of existing studies in this area. Figure 4 displays the distribution of publications by clinical specialties, medical scenarios, sensitive attributes, and publication venues. Because individual studies may address more than one clinical specialty, medical scenario, or sensitive attribute, the counts in these categories are not mutually exclusive. Additionally, we provide the index of available large AI models and datasets for future medical bias research in the Supplementary Information File.
Fig. 4: Distribution of the 92 included publications by clinical specialty, medical scenario, sensitive attribute, and publication venue.
a Number of publications addressing each clinical specialty. b Number of publications addressing each medical scenario: clinical decision support, patient communication, medical documentation, and medical education. c Number of publications examining each sensitive attribute. In (a–c), bars represent publication counts, and the numbers at the ends of the bars indicate the exact counts. Individual publications may cover multiple clinical specialties, medical scenarios, or sensitive attributes; therefore, counts within each panel may exceed 92 in total. d Distribution of publications by venue category: medical and healthcare venues (dark grey; 51 publications), non-medical venues (medium grey; 15 publications), and preprints (light grey; 26 publications). Slice sizes represent the proportions of the 92 publications, and numbers within the slices indicate publication counts.
Medical scenarios distribution
Regarding the application distribution of medical bias, 70 involve clinical decision support, 24 involve patient communication, 9 involve medical documentation, and 7 involve medical education. The predominance of clinical decision support suggests that current research primarily focuses on biases in AI-assisted clinical decision-making, including diagnostic support, triage, risk assessment, and treatment recommendations. This emphasis likely reflects both the clinical importance and the relative technological maturity of decision-support systems, which offer clearer evaluation benchmarks and more measurable outcomes. In contrast, studies on patient communication, medical documentation, and particularly medical education remain comparatively underexplored. One likely explanation is that these scenarios involve more subjective, context-dependent, and interaction-oriented tasks, for which standardized datasets and evaluation metrics are less well established. Consequently, biases in these domains, although potentially more subtle and socially consequential, have received less systematic investigation than biases in clinical decision-making applications.
Clinical specialties distribution
Statistical analysis shows that general healthcare dominates the literature, appearing in 35 of the 92 analyzed papers. This prominence is likely driven by the widespread use of general medical question-answering benchmarks such as MedQA and AMQA, which provide shared evaluation frameworks that facilitate more feasible and comparable bias analyses across models. Psychiatry also receives substantial attention, appearing in 14 papers, followed by cardiology in 13 papers, dermatology and pulmonology each in 12 papers, ophthalmology in 11 papers, and oncology in 10 papers. These concentrations likely reflect both the high clinical importance of these domains and the availability of structured datasets, such as medical imaging repositories, clinical vignettes, and EHR-derived resources, that support quantitative bias analysis. In contrast, several specialties remain comparatively underrepresented. Anesthesiology was examined in 3 papers, and nephrology and gastroenterology were each examined in 2 papers. This uneven distribution highlights an important research gap, as several specialties remain insufficiently examined despite their substantial social and clinical significance.
Sensitive attributes distribution
We observe that sex, race, and age are the three most frequently studied attributes, appearing in 69, 64, and 35 papers, respectively. This pattern aligns with prior surveys on bias and fairness in general AI domains41,105, in which these attributes also dominate. Their prevalence likely reflects both data availability and societal salience: demographic information such as race, sex, and age is routinely recorded in medical datasets, and disparities along these dimensions have long been recognized in healthcare outcomes. Socioeconomic status appears in 23 papers, LGBTQ+ attributes in 16 papers, and language in 13 papers, indicating growing but still limited attention to non-English, multilingual, and structurally disadvantaged healthcare contexts. In contrast, several attributes appear in fewer papers, including culture, disability, housing status, and body size. This imbalance highlights an important gap in the literature. Existing studies primarily focus on well-documented demographic biases, whereas contextual, intersectional, and structural factors that may also contribute to inequities in medical AI systems remain comparatively underexplored.
Venue distribution
As this line of research lies at the intersection of AI, medicine, and healthcare, both medical and non-medical venues serve as important publication outlets. Our analysis shows that 51 studies were published in medical or healthcare-related venues, including medical, clinical, health informatics, and digital health journals. In addition, 15 studies appeared in non-medical venues, including AI, computer science, and interdisciplinary science outlets. Notably, 26 studies were disseminated through preprint platforms, including arXiv, medRxiv, and Research Square. This distribution reflects both the interdisciplinary nature of research on medical bias in large AI models and the rapidly evolving landscape of large AI technologies, which has encouraged researchers to disseminate findings through both traditional publication channels and open preprint repositories.
Mapping between bias type and subgroup
To further connect our scenario-based taxonomy with the existing evidence base, we mapped each medical bias type to the subgroup attributes examined in the included studies. Table 4 shows that most subgroup analyses are concentrated in studies of clinical decision support bias, particularly for race, sex, and age. In contrast, structurally mediated and intersectional attributes, such as disability, insurance status, housing status, culture or nationality, and intersectional identity, are examined far less frequently. This distribution highlights an important gap between the demographic categories most commonly evaluated and the broader forms of social vulnerability that are highly relevant to equitable medical AI systems.
Table 4 Mapping of medical bias, subgroup attributes, and study counts