Table 1 Overview of MedGemma evaluation datasetsMedical text question answering<\/p>\n
Across all text-only biomedical question-answering tasks evaluated, MedGemma demonstrated superior performance to the \u2018baseline\u2019 Gemma 3 model, as well as competitive performance with much larger models in many cases (Table 2<\/a>). Confidence intervals for the results, where calculated, are shown in Supplementary Table 1<\/a>. Acknowledging that frontier models often still outperform smaller open models, these results highlight the clear benefit of domain specialization and position MedGemma as a useful starting point for many downstream applications. This evaluation was based on MedQA, MedMCQA, PubMedQA, MMLU subsets, AfriMed-QA and the OOD MedXpertQA. Notably, MedGemma\u2019s performance of 86.2 on MedQA represented the best performance for an open model at the time of its release, other than the 671-billion-parameter DeepSeek-R1 model (91.0)16<\/a>, and was substantially higher than the next closest well-documented open model, OpenBioLLM 70B (78.2)17<\/a>.<\/p>\n Table 2 Accuracy on medical text benchmarks<\/p>\n The open nature of MedGemma has also enabled additional third-party evaluations. For medical text question answering, the analysis provided in ref. 18<\/a> further demonstrates the performance advantages conferred by MedGemma\u2019s medical domain training relative to Gemma 3. One notable difference was the improved performance of MedGemma relative to Gemma 3 on the MedHallu benchmark evaluation19<\/a>, which asks a model to distinguish between a correct answer and a feasible sounding, but incorrect answer as one way to better understand model reliability. On the established \u2018hard set\u2019 of the MedHallu task, MedGemma 4B scored 0.50 and Gemma 3 4B scored 0.09 (normalized performance score).<\/p>\n Medical image classification<\/p>\n Medical image interpretation is a key contribution of the multimodal MedGemma models. To provide performance insights across a diverse set of medical imaging domains, we used image-based classification tasks in radiology, histopathology, dermatology and ophthalmology. These domains were selected because they are most consistent with training data and intended capabilities of MedGemma. (Of note, addressing classification as a zero-shot generative task may not provide maximum performance as compared to an embedding-based, task-specific classifier, but these evaluations are intended to provide insights into the quality and performance of the underlying image encoder and the overall model capabilities.) Results are summarized in Table 3<\/a> for CXR, histopathology, dermatology, and retina image classification. Notably, MedGemma not only demonstrated substantially superior performance relative to the standard Gemma 3 baselines but also outperformed much larger API-based models.<\/p>\n Table 3 Accuracy on zero-shot medical image classificationOpen-ended medical question answering and clinical reasoning<\/p>\n To further evaluate the quality of model output, generated responses to open-ended medical content questions were evaluated. Model responses to these questions were rated by three physicians per question (see Supplementary Note<\/a> for additional details). This analysis demonstrated a relative decrease in factually inaccurate responses of approximately 17% for MedGemma 4B versus Gemma 3 4B (inaccurate ratings for 18.5% of Gemma 3 responses versus 15.4% of MedGemma responses; Supplementary Fig. 1<\/a>). While the decrease in inaccurate responses for MedGemma suggests a benefit of the medically specialized model, this type of evaluation is subject to potential interrater variability and application-specific considerations, highlighting the importance of use-case-specific fine-tuning and evaluation.<\/p>\n MedGemma is also capable of engaging in open-ended clinical reasoning tasks, a relatively unique capability for a model of its size. We explored this capability using open-ended clinical vignette prompts and examined the MedGemma text output through clinical expert review. Overall, MedGemma effectively summarized key components of the scenarios and identified relevant management recommendations. Examples of these tasks, along with clinical commentary, are shown in Supplementary Note\u2014Medical reasoning examples<\/a>.<\/p>\n Medical visual question answering<\/p>\n To evaluate medical VQA, we used the often-reported benchmarks SLAKE20<\/a> and VQA-RAD21<\/a>. While different implementations for evaluating these benchmarks can make direct comparisons across literature challenging, we report direct comparisons of MedGemma to a number of additional models where possible as one component of demonstrating its strong baseline capabilities across modalities. Note that to avoid train\u2013test contamination in VQA-RAD, we used the train and test splits described in refs. 9<\/a>,22<\/a>. These results are summarized in Extended Data Table 2<\/a>. Again, MedGemma outperformed the standard Gemma 3 model and overall performed similarly to the much larger API-based models. LLaVA-Med and BiomedGPT-B are also included in the results table using the available reported metrics23<\/a>,24<\/a>, noting that these models were involved in supervised fine-tuning (SFT) for the specific tasks and datasets. In addition to these quantitative results, we show several illustrative examples of open-ended visual question answering in Fig. 2<\/a>.<\/p>\n Fig. 2: Examples of open-ended multimodal question answering with MedGemma 4B. Prompts that include both text and an image are shown along with the MedGemma 4B response. Assessments of the response by US board-certified clinicians are shown at the bottom of each section. X-ray image sourced from the NIH Clinical Center (https:\/\/nihcc.app.box.com\/v\/ChestXray-NIHCC\/folder\/37178474737<\/a>), reproduced from ref. 54<\/a>; dermatology image sourced from the Skin Condition Image Network dataset (https:\/\/github.com\/google-research-datasets\/scin<\/a>), reproduced from ref. 55<\/a>; and histopathology image sourced from The Cancer Genome Atlas56<\/a>.<\/p>\n It is important to clarify that we did not compare MedGemma with specialized models in the literature for which these metrics were calculated using a discriminative, embedding-based approach. Such approaches can be informative but use the training set to define the set of possible text answers, and thus represent the task as a limited-vocabulary classification task rather than an open-ended VQA task. Similarly, while LLaVA-Tri25<\/a> also reports strong performance, the reported methodological details for these metrics are insufficient to support a direct comparison. Lastly, several other models have reported strong performance on these VQA benchmarks, such as HuatuoGPT-Vision26<\/a> and PMC-Cambrian-8B-AN27<\/a>, with accuracy results reported in aggregate across open and closed questions. Given the challenge of comparing metrics even for the same benchmarks, we interpret the reported results for MedGemma on these VQA tasks as an important component towards demonstrating strong baseline performance across modalities.<\/p>\n CXR report generation<\/p>\n Results of performance for CXR report generation are shown in Table 4<\/a>. These results are based on the MIMIC-CXR dataset and the RadGraph F1 metric28<\/a>. The pretrained checkpoint of MedGemma 4B demonstrates results similar to or better than other top-performing models, which is particularly notable given the comparator models are either larger or specialized only for this task. This shows the versatility of the Gemma architecture in adapting to complex medical use cases and enabling MedGemma. In addition, expert human evaluation demonstrated that 68% of generated reports from normal studies and 49% of generated reports from abnormal studies were rated as equal to or superior to the original reports (Extended Data Fig. 1<\/a>). This corresponds to 81% of generated reports rated as likely to result in the same or superior clinical decisions as compared to the original reports. This represents superior performance to the larger Med-Gemini model, for which 73% of generated reports were rated as resulting in the same or superior clinical decisions as compared to the original reports9<\/a>.<\/p>\n Table 4 Report generation on the MIMIC-CXR dataset<\/p>\n Again, the open nature of MedGemma has enabled third-party evaluations. Specifically, for CXR report generation, the ReXrank leaderboard (ReXrank.ai) provides standardized evaluation across several models, datasets and metrics for CXR interpretation. Notably, MedGemma 4B demonstrates the highest Generative Radiology Report Evaluation and Error Notation (GREEN) score29<\/a> on the ReXGradient dataset30<\/a> (GREEN score 0.566) and the IU-Xray set31<\/a> (GREEN score 0.724), with middle-of-the-leaderboard performance on the MIMIC-III and CheXpert Plus datasets. MedGemma also demonstrates top-three performance on the ReXVQA medical VQA evaluation32<\/a>. (ReXrank.ai, accessed 9 February 2026). While these results are consistent with strong baseline performance for MedGemma on CXR report generation and VQA, including on datasets not seen during training, the thorough ReXrank evaluation also notably illustrates that different models can excel on different metrics and datasets. As such, users can consider the aspects and metrics most important for their use case when selecting a model for further development and validation.<\/p>\n CXR anatomical structure localization<\/p>\n Although specific optimizations were not performed for this task, because anatomical bounding boxes were included in the training data for MedGemma 27B, we evaluated the localization of anatomical structures using the intersection over union (IoU) metric on the test set of the Chest ImaGenome dataset33<\/a>. The findings are included in Supplementary Table 9<\/a> for reference, with IoU values of 0.16 for MedGemma 27B, 0.08 for baseline Gemma 3 27B, 0.03 for MedGemma 4B and 0.09 for Qwen3-VL 4B. These results demonstrate improved performance on this task when relevant training data are added, but indicate there is still substantial room for optimization and improvement of this capability.<\/p>\n Medical agentic behavior<\/p>\n To provide initial insights into the capabilities of MedGemma in the setting of use cases beyond single-turn question answering, we also evaluated MedGemma in an agentic role using the AgentClinic benchmark15<\/a>. In this task, the model functions as a \u2018physician agent\u2019 in simulated clinical encounters, performing tasks such as patient history-taking, exam interpretation and diagnosis under uncertainty to reach the correct answers. Using the text-only AgentClinic-MedQA and AgentClinic-MIMIC-IV environments (with GPT 4o for the patient agent and measurement), MedGemma 27B exceeded the performance of human physicians on AgentClinic-MedQA and approached the performance of much larger models on both datasets. Results are summarized in Extended Data Table 3<\/a>. Notably, we found that the smaller Gemma 3 4B and MedGemma 4B models were poorly suited to this task, demonstrating difficulty following system instructions within the AgentClinic agentic framework.<\/p>\n General purpose benchmarks<\/p>\n Given the limitations and poor performance that many medically specialized models exhibit when faced with nonmedical tasks, which may critically interweave with medical applications, we evaluated possible trade-offs of specialization. Models were evaluated on general language model benchmarks including MMLU Pro, Global MMLU Lite and MMMU datasets, with performance comparisons between MedGemma and Gemma 3 variants shown in Extended Data Table 4<\/a>. MedGemma demonstrated only minor decreases in performance relative to the general models of the same size, further suggesting its potential utility for downstream applications that require both specialized and generalized capabilities, such as instruction following or diverse user interactions.<\/p>\n Observed themes<\/p>\n Several key themes emerged from our analysis. While larger, more computationally expensive models often performed better, medically specialized models such as MedGemma exhibited a distinct performance-efficiency advantage when their substantially smaller size was taken into account. This size-to-performance benefit was especially pronounced in medical vision-based tasks, where MedGemma surpassed the performance of substantially larger general-purpose models. Notably, there was a 500-fold difference in computational cost between MedGemma 4B and the most expensive comparator model, a crucial consideration for practical application development, where both development and compute usage are constraints.<\/p>\n We also observed that performance on older, established benchmarks tended to improve with newer models. While this observation reflects genuine advancements in model capabilities, it also raises the possibility of test data leakage, as these benchmarks are publicly available and frequently used for model development and evaluation.<\/p>\n Fine-tuning MedGemma enables robust task-specific adaptation<\/p>\n While the MedGemma models achieved strong baseline performance, these models are intended as a useful starting point for specialized task adaptation (for example, specific styles, specific classification tasks, or other user needs).<\/p>\n We conducted fine-tuning experiments using three different datasets (not previously included in MedGemma training) and using either MedGemma 4B or Gemma 3 4B as the base model. The results are shown in Extended Data Fig. 2<\/a> and generally demonstrate an increase in performance with fine-tuning and with higher performance for MedGemma 4B as compared to Gemma 3 4B. This performance advantage for MedGemma was particularly notable when using only 10% of the available data for a given task, supporting the notion of MedGemma as a useful starting point for data-efficient fine-tuning. We also note that for SIIM-ACR, Gemma 3 collapses to predicting the majority class, whereas MedGemma avoids this collapse, likely due, at least in part, to being pretrained on radiology data. While these results are provided to demonstrate illustrative comparisons of fine-tuning for classification tasks using a common SFT approach, specific fine-tuning methods and prompt engineering may benefit from further use-case-specific optimization.<\/p>\n Fine-tuning of MedGemma 4B with reinforcement learning (RL) was also performed to demonstrate further optimized performance for CXR report generation, as a task that was also included in MedGemma training. The resulting fine-tuned model achieved a RadGraph F1 score of 30.3 (for findings plus impression) on MIMIC-CXR, representing state-of-the-art performance for this specific metric and dataset, as shown in Table 4<\/a>.<\/p>\n We also performed RL and evaluation experiments on EHRQA, a synthetic dataset specifically focused on longitudinal outpatient electronic health record data. Fine-tuning MedGemma 27B Text using RL on the EHRQA training set resulted in improved performance, with the greatest gains observed for question categories requiring reasoning across interdependent records. These results further support the potential of fine-tuning MedGemma for effective use in task-specific electronic health record applications. Additional details about the dataset and experiments are described in the Supplementary Note<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"MedGemma performance across medical text and imaging benchmarks MedGemma performance was evaluated across several medical text and medical…\n","protected":false},"author":2,"featured_media":622613,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[34],"tags":[6471,6472,2294,134,3186,527,6474,45530,3060,6473,6475,6476,111,139,69],"class_list":["post-622612","post","type-post","status-publish","format-standard","has-post-thumbnail","category-healthcare","tag-biomedicine","tag-cancer-research","tag-general","tag-health","tag-health-care","tag-healthcare","tag-infectious-diseases","tag-medical-imaging","tag-medical-research","tag-metabolic-diseases","tag-molecular-medicine","tag-neurosciences","tag-new-zealand","tag-newzealand","tag-nz"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/622612","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=622612"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/622612\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/622613"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=622612"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=622612"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=622612"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"Fig.](\"https:\/\/www.newsbeep.com\/nz\/wp-content\/uploads\/2026\/10\/41591_2026_4626_Fig2_HTML.png\")
<\/p>\n