Large language models are powerful electronic health record encoders

EHRSHOT database and prediction task

The EHR data used in our experiments are from the EHRSHOT benchmark for few-shot evaluation of clinical EHR prediction45. We obtained version 2.1 of the dataset, which is accessible via gated access under a research data use agreement. This dataset comprises longitudinal records for 6739 patients, 921,499 visits, and 41,661,637 clinical events collected between 1990 and February 8, 2023 (Table 1). Each clinical event is linked to a specific patient and includes information such as start time, end time, a semantic code, a value, unit, visit ID, and the corresponding Observational Medical Outcomes Partnership (OMOP) source table. We used the official ehrshot-benchmark GitHub repository as a starting point to design our experiments45, enabling us to build on existing functionalities and facilitate comparisons with prior methods. The benchmark uses the framework for electronic medical records (FEMR) GitHub repository, which provides Python classes for efficient loading and processing of EHR data. All extensions and experiments conducted for this paper are publicly available via our GitHub repository: https://github.com/stefanhgm/ehrshot-benchmark. The EHRSHOT benchmark defines a rigorous evaluation including 15 clinical prediction tasks categorized into four groups: operational outcomes, anticipating lab-test results, assignment of new diagnoses, and anticipating chest X-ray findings45. Task labels are derived from clinical events, so a single patient can contribute multiple labels per task, resulting in substantial variation in task-specific sample sizes (Table 2). For instance, frequent events such as laboratory tests yield many more examples than rarer events such as incident diagnoses. The benchmark focuses on analyzing model performance in a few-shot setting, which is particularly relevant for large pretrained foundation models14. Specifically, for k ∈ {1, 2, 4, 8, 12, 16, 24, 32, 48, 64, 128}, the protocol uses k positive and k negative training examples and k positive and k negative validation examples sampled with replacement to train and tune supervised classifiers. If fewer than k positive or negative training examples were available, the next larger shot size was used with resampling. Using the full data for training and validation, which are often unbalanced, is included for all tasks. Testing is always performed on the full set of available examples for each task. The classifiers evaluated within the EHRSHOT framework include logistic regression, random forests, and GBMs57. Performance is reported using the AUROC, the AUPRC, and the Brier score. For few-shot settings, results are averaged over multiple runs with different random seeds, and variability across runs is summarized using the standard deviation of the test set performance45. For each individual task, uncertainty is additionally estimated via bootstrap resampling of the test set to obtain standard deviations and percentile-based 95% confidence intervals. For task groups, mean performance is computed by averaging across tasks, and approximate 95% confidence intervals are derived from the uncertainties of the individual task estimates by combining their variances and applying a normal approximation. An overall macro average is computed by averaging across task groups, with corresponding uncertainty estimated analogously.

EHR text serialization

To apply LLM embedding models to EHR data, we serialized each patient record into task-agnostic text. Following EHRSHOT, we included all events occurring before the label time and truncated inputs to 8192 tokens (~32,000 characters) due to computational constraints (see section “Computational setup and runtime”). Each clinical event in EHRSHOT is represented by a semantic identifier of the form “ontology/code”. We resolved these identifiers to text using the provided vocabularies and custom mappings. To assess ontology coverage, we analyzed events from 200 patients across the operational-outcomes and new-diagnoses task groups (2968 labels). We observed events from the following ontologies: Logical Observation Identifiers Names and Codes (LOINC), SNOMED, RxNorm, CPT4, Domain, CARE_SITE, RxNorm Extension, Medicare Specialty, ICD10PCS, CMS Place of Service, Cancer Modifier, ICD9Proc, CVX, ICDO3, HCPCS, OMOP Extension, and Condition Type. Codes from CPT4, CARE_SITE, ICD10PCS, Cancer Modifier, CVX, and ICDO3 were not fully resolved by the provided resources. We parsed UICC cancer stages from Cancer Modifier codes and added manual mapping files for CPT4, ICD10PCS, and CVX. We excluded CARE_SITE and ICDO3 because we could not map them to useful descriptions.

We used a simple newline-separated list of event descriptions with minimal preprocessing to emphasize the embedding model’s ability to interpret raw clinical content. To remain within the token budget, we kept only the most recent occurrence of each code. This default serialization, therefore, corresponds to a list of unique codes per patient record with code-to-description mapping and deduplication as the only mandatory preprocessing steps. When available, we appended values and units in brackets, and values of the datatype float were rounded to two decimals. We also evaluated variants that kept the first occurrence of each code and that added date and time information, but these did not improve performance (Table 6).

Testing alternative EHR text serializations

To evaluate whether more structured inputs improve performance, we implemented an alternative Markdown-based serialization, a common LLM input format58. We included visit dates and the number of days before prediction time, normalizing all dates relative to January 1, 2024, the prediction reference time. The serialization started with demographics, and birthdates were converted to age in years. Because 65% of events were LOINC-coded time series, we aggregated LOINC events. Using the same 200-patient subset as above, we selected the 24 most frequent concepts across vital signs, body metrics, and laboratory values and merged synonymous codes (Table S2). For each concept, we reported the last three values and removed implausible measurements. We additionally added default units and categorical interpretations (low, normal, high) based on standard ranges (Table S2). Following the aggregated data, a summary of all visits was included to address the potential truncation of older visits. Events not associated with visits were then presented, using the same aggregation logic to display the last three values where applicable. Finally, a detailed reverse-chronological presentation of all visits was included, with events categorized into conditions (SNOMED, Visit, Cancer Modifier, CVX, HCPCS), medications (RxNorm, RxNorm Extension), and procedures (CPT4, ICD10PCS, ICD9Proc).

Additionally, we converted the Markdown serialization into JSON, XML, and YAML using standard Python libraries to assess format effects under a fixed token budget. Because these formats have different overhead, the amount of clinical content retained can vary for the same token limit.

LLM embedding models and baselines

We evaluated three LLM embedding models: Qwen3-Embedding-8B (Qwen3-Emb-8B)39,44, GTE-Qwen2-7B-Instruct (Qwen2-Emb-7B)38,43, and LLM2Vec-Llama-3.1-8B-Instruct (Llama3.1-LLM2Vec-8B)40,49, based on state-of-the-art decoder-only LLMs. We selected these models for their ability to handle the 8192-token EHR serializations used in our experiments. For comparison, we also tested smaller variants of Qwen3-Emb-8B and Qwen2-Emb-7B. All LLM-based models received task-specific instructions prepended to the EHR serialization (see section “ Instructions for LLM embedding models”), and we used each model’s default embedding configuration. For Qwen-based embedding models, we used the last-layer hidden state of the final [EOS] token as the patient embedding38,44. For LLM2Vec models, we applied mean pooling over the last-layer hidden states corresponding to the EHR serialization, excluding the instruction tokens from pooling40. Importantly, the instruction still conditions the internal representation and thus influences the resulting embedding.

As additional baselines, we included commonly used encoder-only embedding models with shorter input limits of 512 tokens. To apply them to 8192-token inputs, we split each input into up to 16 chunks of 512 tokens and evaluated mean pooling and concatenation of these chunk embeddings to obtain a single representation. For the biomedical fine-tuned encoders MedBERT and BioClinicalBERT, we also tested the MEME method to encode domain-specific data separately with a logistic regression classification head to follow our study setup34. Overall, this yielded 14 encoder-only baseline configurations for long-context EHR prediction. Encoder-only models did not use task instructions. Below is an overview of all models.

Qwen3-Embedding-8B/4B/0.6B is a model family built on the Qwen3 foundation LLMs39 and trained as instruction-aware embedding models via a multi-stage recipe44. All variants use decoder-only Transformers with causal attention and produce embeddings from the last-layer hidden state of the final [EOS] token (see section “Instructions for LLM embedding models”). The 8B and 4B models have 36 layers, while the 0.6B model has 28 layers, and all support a context window of up to 32,000 tokens. By default, the embedding dimensionalities are 4096 (8B), 2560 (4B), and 1024 (0.6B), with support for flexible output dimensions. The base model Qwen3 was trained on a multilingual corpus of approximately 36 trillion tokens39. Training of the embedding models comprises large-scale weakly supervised pretraining on synthetic query–document pairs, followed by supervised fine-tuning on labeled and filtered synthetic data, optimizing a contrastive learning objective. Finally, different checkpoints are merged to improve robustness44.

Qwen2-Embedding-7B/1.5B is based on GTE-Qwen2-7B-Instruct and GTE-Qwen2-1.5B-Instruct38. These models use decoder-only Transformers (7B: 28 layers, 28 heads, hidden size 3584; 1.5B: 28 layers, 12 heads, hidden size 1536). They are first trained with autoregressive next-token prediction and then converted to embedding models via the general text embedding (GTE) procedure43, which replaces the causal mask with BERT-like bidirectional attention during embedding extraction. Contrastive learning is applied using a mixture of private datasets to enhance embedding performance. The models also incorporate instructions tailored for embedding tasks and support a context size of up to 32,000 tokens.

Llama3.1-LLM2Vec-8B is built upon the Llama-3.1-8B-Instruct LLM49, which has a decoder-only Transformer architecture with 32 layers, 32 attention heads, and a hidden size of 4096 (Hugging Face Identifier: McGill-NLP/LLM2Vec-Meta-Llama-31-8B-Instruct-mntp-supervised). Initially trained for next-token prediction, it was converted to an embedding model using the LLM2Vec method40. This method adds bidirectional attention and fine-tunes the model with supervised contrastive learning on embedding tasks. Fine-tuning used curated data from the public E5 dataset59,60, containing ~1.5 million entries. The model supports task-specific instructions and a context size of up to 128,000 tokens.

DeBERTa v3 base/large are encoder-only Transformer models designed for token embeddings61. They improve upon their predecessor by replacing the masked language modeling objective with replaced token detection and using gradient-disentangled embedding sharing. We evaluated the base variant (12 layers, 12 attention heads, 768 hidden size) and the large variant (24 layers, 12 attention heads, 1024 hidden size), with parameter counts of 183M and 434M, respectively (Hugging Face Identifier: microsoft/deberta-v3-{base,large}).

BERT base/large are well-established text embedding models using an encoder-only transformer trained with the masked language modeling objective15. We included both the base (12 layers, 12 attention heads, 768 hidden size, 110M parameters) and large (24 layers, 16 attention heads, 1024 hidden size, 340M parameters) variants as benchmarks (Hugging Face identifier: google-bert/bert-{base,large}-uncased). While not state-of-the-art, BERT models remain widely used in embedding tasks.

BioClinicalBERT builds on BERT-Base, further fine-tuned on biomedical62 and clinical data54. It is a widely adopted embedding model for medical text and was included as a baseline for comparison (Hugging Face identifier: emilyalsentzer/Bio_ClinicalBERT).

MedBERT builds on BioClinicalBERT through continued domain pretraining on heterogeneous biomedical corpora (N2C2, BioNLP, CRAFT, and biomedical Wikipedia) and was originally introduced for biomedical NER63. We include it as a domain-adapted baseline for comparison (Hugging Face identifier: Charangan/MedBERT).

CLMBR-T-Base is a specialized EHR foundation model trained on 2.57 million de-identified EHRs from Stanford Medicine with autoregressive next-code prediction13,45. With the estimated 706 mean clinical events per patient45, this leads to ~1.8 billion clinical events for training. CLMBR-T-Base has 12 transformer layers and a hidden dimension of 768 (Hugging Face Identifier: StanfordShahLab/clmbr-t-base). The model has 141M parameters and allows for a context window of 496 codes. CLMBR-T-Base has demonstrated improvements over count-based baselines across a variety of clinical prediction tasks45. It serves as the main baseline for our experiments to test specialized EHR models against general-purpose text embedding models for representing EHR records.

LLM embedding model and CLMBR-T-Base are a model combination used to test whether the LLM embedding models and the EHR foundation model learn orthogonal information. To this end, we simply appended both embeddings. The resulting embeddings have dimensions of 4864 for Qwen3-Emb-8B.

Count-based models have proven to be strong baselines for EHR prediction tasks6,12,13. All EHR events of a patient are encoded in a single vector, where each entry represents the number of occurrences of a medical concept. We used the count-based baseline introduced in ref. 45, which uses counts for all clinical events that occurred in a patient’s timeline prior to the prediction time and extends this approach with ontology expansion, enriching the vectors with parent and child concepts. We also included three extensions of the above count baseline implemented in the Electronic Medical Records (FEMR) toolkit13, using (1) time binning of 0–30, 30–180, 180–365, and 365+ days13, (2) numeric values partitioned into deciles6, (3) string values6, and a combination of all three (Table S6).

Based on the embeddings or count vectors generated by the methods described above, a classification head was trained and validated for each prediction task. For the embedding models, we used a logistic regression head as the default. For the count-based model, we used a GBM57 as the primary classifier because it performs better for high-dimensional count features (Table S4). For completeness, we also report count-based results with a logistic regression head. We adopted the parameter tuning of the classification heads from the EHRSHOT benchmark to ensure comparability45.

Instructions for LLM embedding models

The Qwen2, Qwen3, and LLM2Vec models use instruction-tuned embeddings. We therefore added simple prompts for each prediction task based on their respective templates. For instance, for the prediction of anemia, we added: “Given a patient’s electronic healthcare record (EHR) as a newline-separated list, retrieve relevant passages that answer the query: has the patient anemia”. The existing EHRSHOT benchmark encodes EHRs for the same patient and identical prediction times only once for efficiency. To support task-specific instructions, we changed this behavior and encoded each (patient, task, prediction time) instance, resulting in 1,161,412 instead of 406,379 EHRs and longer processing times. The difference between the 1,161,412 labels used in our experiments and the total number of 1,178,665 labels (Table 2) arises because some labels share the same task and prediction time and are therefore merged. For experiments in UKB, we designed analogous instructions. We list all instructions in Table S3 and perform ablations to test their effect.

Fine-tuned LLM embedding and LLM decoder models

The main approach evaluated in this paper used LLM embedding models that encode the EHR serialization into a fixed-length vector prior to classification. For comparison, we also tested LLM decoder models that generate text whose token probabilities are used for classification. We further assessed the effect of fine-tuning for both approaches. For comparability, all encoder, decoder, and decoder-ICL experiments used the same default list-based EHR serialization and the same task-specific instructions. In contrast to the other experiments, inputs were truncated to 4096 tokens due to computational restrictions.

For the encoder setting, we fine-tuned Qwen3-Emb-8B39,44 with LoRA adapters64 applied to the attention and MLP projection modules (q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj). The encoder was paired with a lightweight classification head consisting of a dropout layer and a linear projection, and optimized end-to-end with cross-entropy loss. Inputs were created by prepending the task-specific instruction to the serialized EHR and using tokenization with left padding to preserve recency under truncation.

For the decoder setting, we adapted the instruction-tuned causal language model Qwen3-8B (Hugging Face identifier: Qwen/Qwen3-8B) with LoRA adapters on the same projection modules. The decoder received a chat-style prompt containing the task instruction and the serialized EHR (Table S3), and was trained to predict the literal next token, Yes or No. At evaluation time, we scored each example by aggregating the probability mass assigned to single-token variants of Yes and No, including capitalization and punctuation variants, and then normalizing to a two-way probability.

We also evaluated decoder ICL without weight updates. These prompts used the same task-specific instruction and deterministic balanced example selection. We evaluated 0-, 2-, 4-, and 6-shot ICL only; larger ICL settings were not computationally feasible. To limit prompt growth, we applied separate 4096-token caps to the rendered ICL examples block and to the base prompt containing the target patient record.

For both fine-tuned model families, we kept the optimizer and most training settings fixed and tuned only the main adaptation hyperparameters. The tuning grid covered learning rates of 5 × 10−5, 10−4, and 2 × 10−4, LoRA ranks of 8, 16, and 64, and LoRA dropout values of 0.0, 0.05, and 0.1, yielding 27 configurations per model family. We fixed LoRA α = 32, warmup ratio 0.03, a maximum of 20 epochs, AdamW optimization via the Hugging Face Trainer, cosine learning-rate decay, early stopping with patience 5, gradient checkpointing, and an effective batch size of 8. Per-task tuning across all EHRSHOT subtasks was computationally infeasible. We therefore performed tuning on the operational-outcomes and new-diagnosis tasks only (guo_* and new_*) at k ∈ {8, 16} using validation AUROC as the selection criterion, and then chose one shared setting per model family based on mean validation AUROC across those tasks (Table S13). This selected an encoder setting with learning rate 5 × 10−5, LoRA rank 8, and dropout 0.1, and a decoder setting with learning rate 2 × 10−4, LoRA rank 8, and dropout 0.05. Using these fixed settings, we reran the full fine-tuning matrix across all EHRSHOT subtasks, k ∈ {1, 2, 4, 8, 16, 32, 64, 128}, and five replicates. For both model families, evaluation computed AUROC, AUPRC, and Brier score on the held-out test set together with 1000 patient-level bootstrap replicates to estimate standard deviations and 95% confidence intervals.

Existing methods using language models for EHR prediction

Two broad approaches have emerged for leveraging language models with structured EHR data: generation-based prediction and embedding-based prediction. Early work on generation-based approaches converted structured claims into short text snippets and queried encoder-decoder models with a 512-token context limit, showing improvements in few-shot regimes (16–64 examples) for end-of-life, surgery, and length-of-stay tasks26. Subsequent studies explored larger LLMs, prompt design, and fine-tuning strategies on serialized EHR inputs, typically treating prediction as text generation or calibrated scoring27,28,29,30,31,32. One variant constrains the model’s output space to medical codes to better align free-text reasoning with code-based clinical labels, thereby bridging the gap between general-purpose language models and EHR-specific foundation models65.

Embedding-based methods instead extract fixed-dimensional representations for downstream classifiers33,34,35,36. Gao et al. provide a systematic comparison across serialization formats, LLMs, and embedding strategies on MIMIC-III and a private 660-patient clinical-deterioration dataset, reporting that LLM embeddings paired with a prediction head can be competitive in some settings, while raw numerical features remain strong baselines33. They also evaluate an encoder-based approach producing Yes and No tokens in the zero-shot setting, but find no discriminatory ability for Mistral-7B-Instruct and Llama3-8B-Instruct. The GRASP framework shows that combining LLM-derived embeddings of medical codes with transformer predictors improves cross-dataset generalization36. The DeLLiriuM method introduced in ref. 35 also evaluates embeddings derived from different LLMs for delirium prediction and performs detailed model introspection via a SHAP analysis. Most similar to our work is the MEME method, which encodes different EHR modalities such as vitals, medications, and diagnoses into separate embeddings and fuses them with a self-attention layer34. This framework uses the relatively small and efficient MedBERT language model for embedding creation and reports superior performance to GPT-434.

Our study follows the embedding paradigm but differs from prior work in three ways. First, we use LLMs explicitly adapted for embedding via contrastive learning. Second, we operate with substantially longer inputs (up to 8192 tokens), which is important for longitudinal EHR serialization, where earlier studies often processed up to 3076 tokens33 or relied on chunking34. Third, we evaluate on large public longitudinal cohorts and standardized benchmarks from EHRSHOT and UKB. We also include a MEME-style baseline with a linear head in place of self-attention due to data sparsity34. Finally, consistent with ref. 33, we explicitly compare embeddings to direct LLM outputs and examine parameter-efficient fine-tuning to assess robustness and practical trade-offs (see section “Fine-tuned LLM embedding and LLM decoder models”).

Computational setup and runtime

All experiments were conducted on the Charité high-performance cluster offering different GPU setups. Due to the larger memory requirements of the LLMs, we used a smaller batch size for inference compared to encoder-based models and CLMBR-T-Base. For the Llama3.1-LLM2Vec-8B model, runtime errors occurred during multi-GPU experiments with the full dataset. These issues were resolved by splitting the data into smaller batches, which introduced additional overhead. Additionally, we optimized the LLM2Vec code by removing an initial word-boundary token-limit routine that iteratively re-tokenizes to ensure the final cut occurs at a word boundary, with minimal effect on performance, limited to a potentially incomplete trailing word. To quantify runtime differences, we measured the wall-clock encoding time for the EHRSHOT benchmark on the same 8-GPU Nvidia H200 cluster for CLMBR-T-Base, LLM embedding models, and encoder-based models (Table S12). We tried to use the maximum possible batch sizes. Encoder-based models used 512-token chunks of the up to 8192-token inputs, which substantially increased their runtime. On the UKB, encoding all 387,464 UKB patients for one task with a maximum context length of 8192 on the same H200 cluster took 4:46:34 hours for Qwen3-Emb-8B, 3:49:21 h for Qwen2-Emb-7B, and 3:55:12 h for Llama3.1-LLM2Vec-8B.

Performance results on EHRSHOT prediction tasks and few-shot setting

Following the EHRSHOT benchmark, we evaluated all models across 15 prediction tasks under various few-shot settings. The benchmark includes a modular pipeline designed to execute key tasks, with the flexibility to optionally use a Slurm cluster for distributed execution. Running all steps within this pipeline ensures full reproducibility of results. Step four of the pipeline, which generates EHR representations with CLMBR-T-Base and the count-based model, was extended to incorporate our method for creating language model-based EHR representations. This adaptation allowed the reuse of significant portions of the existing code, including the task evaluation framework. Additionally, we implemented new functionality for EHR serialization and slightly modified other steps of the benchmark to accommodate our experimental setup. For instance, the label creation process was adjusted (step three) to enable task-specific instructions for the LLM embedding models. All modifications have been documented and can be tracked in our public GitHub repository.

External validation on UK Biobank

External validation was performed using data from the UKB, a large-scale prospective cohort study comprising 502,489 UK participants recruited between 2006 and 2010, with a median follow-up of 13.8 years (Table 1). We used linked EHR data from primary care (General Practitioner) and secondary care (Hospital Episode Statistics), providing information on diagnoses, procedures, and prescriptions. We aimed to closely replicate the experimental setup of the EHRSHOT benchmark by using predefined splits of training, validation, and test data (Table S1), defining tasks and task groups accordingly, and applying the same few-shot analysis, hyperparameter tuning scheme, and statistical evaluation.

Initial data preprocessing, including cleaning, feature extraction, missing-value imputation, and endpoint selection, followed the methodology described in ref. 46. All health records were mapped to the OMOP CDM using mapping tables provided by the UKB, SNOMED International, and the OHDSI community for mapping concepts from provider- and country-specific non-standard vocabularies to OMOP standard vocabularies. Participants, not having full demographic information, were excluded. Participants lacking any recorded General Practitioner or Hospital Episode Statistics events either before or after their recruitment date were excluded, resulting in a validation cohort of 387,464 individuals. Diagnostic codes were mapped to Phecodes X66,67 primarily for standardized endpoint definition and cohort selection. To avoid redundancy with medical codes used as features, the Phecodes were excluded during the creation of patient representations. Due to significant challenges in mapping and harmonizing UKB laboratory values68, and to ensure comparability across models and tasks, medical codes of laboratory values were used without numerical values46. The final feature set comprised conditions (SNOMED, CVX), medications (RxNorm), and procedures (SNOMED).

We defined the following tasks for the UKB: (1) prediction of all-cause hospitalization within the next year (operational outcomes), (2) prediction of all-cause mortality (mortality prediction), and (3) prediction of incident diagnoses for a set of selected conditions (assignment of new diagnoses). The selection of diseases for the assignment of the new diagnoses task group largely followed46, focusing on common conditions, diseases lacking established risk stratification tools, and specific cardiovascular conditions. From the initial 24 endpoints proposed in ref. 46, we treated all-cause mortality as a separate task, leading to a total of 25 tasks. For the assignment of new diagnoses and mortality tasks, patients with a diagnosis of the respective endpoint recorded prior to and on the day of their UKB recruitment date were excluded. This exclusion was not applied to the hospitalization task due to the high incidence of hospitalization events before the recruitment date (Table S1). In the external validation on UKB, each patient had only one prediction date, which was marked by their recruitment date. Analogously to the main experiments, we used a simple EHR list serialization of the most recent occurrence of each medical code, with birth and race events added at the time of birth.

The pretrained CLMBR-T-Base model operates with a fixed vocabulary of 26,249 unique codes. To use this model, we mapped the 50,702 unique medical codes (SNOMED CT, RxNorm, CVX) present in our processed UKB cohort to the CLMBR-T-Base vocabulary. This mapping followed steps similar to those implemented in the FEMR package, involving direct code matching where possible, supplemented by indirect mapping via the OHDSI ATHENA vocabulary using “Maps to” relationships and inclusion of ancestor concepts. This process constituted a major effort and required approximately two weeks to complete. Overall, 7969 (16%) unique UKB codes were successfully mapped to the CLMBR-T-Base vocabulary (in the format ontology/code), which were responsible for 25% of medical events in the UKB. The relatively low mapping coverage can be attributed to differences in underlying hospital systems and the distinct purposes of the datasets. Unmapped codes were excluded for CLMBR-T-Base. Additionally, UKB ethnicities were converted to the ethnicity groups used by CLMBR-T-Base. The final data of mapped medical events, birth date, ethnicity, and visits were converted into the MEDS standard69. For transforming patient information into embeddings, we mainly followed the code provided by FEMR, primarily using the convert_patient function with minor modifications to enable batch processing. To disentangle the effects of vocabulary coverage from generalization capabilities, we performed a sensitivity analysis on UKB by restricting Qwen3-Emb-8B to CLMBR-T-Base-mappable codes. We performed the same few-shot experiments and statistical analyses as for the main experiments to compare Qwen3-Emb-8B with all UKB codes, Qwen3-Emb-8B restricted to CLMBR-T-Base-mappable codes, and CLMBR-T-Base (Table 4, Fig. S13, and Table S17).

Analogously to the best count-based model for EHRSHOT, the count model for the UKB included ontology expansion45, increasing the number of unique codes from 51,677 to 69,850, and time binning of 0–30, 30–180, 180–365, and 365+ days13. In contrast to the EHRSHOT experiments, the count-based model did not include string or numeric values because the UKB did not provide them. We also added normalized age at prediction time and coded sex as additional features. Lastly, we evaluated the best encoder-based language model from the EHRSHOT experiments, BioClinicalBERT, on the UKB.

Statistical testing

To assess the significance of performance differences between the best LLM embedding model, Qwen3-Emb-8B, the EHR foundation model CLMBR-T-Base, the encoder-only model BioClinicalBERT, and the count-based baseline, we performed statistical tests for per-task performance on EHRSHOT and the UKB. We evaluated three training regimes: a very few-shot setting with 8 positive and 8 negative examples, a few-shot setting with 64 positive and 64 negative examples, and training on all available data. We excluded tasks with insufficient training data for the 64-shot setting. Few-shot experiments with five-fold cross-validation already evaluate learning efficiency and robustness under limited training data, whereas the statistical tests assess whether the final trained models differ significantly in population-level task performance. Statistical significance of AUROC differences between models was assessed using a paired, patient-level bootstrap on the held-out test set with 10,000 bootstrap replicates. For each shot setting, we used all evaluation replicates by averaging predicted probabilities across replicates for each test example before computing the bootstrap statistic. For the CheXpert task, which comprises 14 binary sub-tasks, the bootstrap statistic is the macro-averaged AUROC, defined as the mean of per-sub-task AUROCs computed within each bootstrap replicate, matching the macro-average metric reported in the main results. We report 95% percentile confidence intervals and two-sided p-values computed from the bootstrap ΔAUROC distribution. For each shot setting, p-values were adjusted jointly across all tasks and baseline comparisons using Holm’s procedure to control the family-wise error rate. This corresponds to a correction over 45 hypotheses per shot setting for 15 tasks and three model comparisons on EHRSHOT, and 75 (8-shot, all data) and 60 (64-shot) hypotheses for the UKB (Tables S5 and S15). We also performed an analogous statistical test for the sensitivity analysis of Qwen3-Emb-8B restricted to CLMBRT-Base-mappable codes on the UKB (Table S17).

Ablation studies of EHR serialization

To better understand the contribution of various components in the EHR serialization process to the performance of the LLM embedding models, we conducted a series of ablation studies. These ablations used Qwen3-Emb-8B with the default list serialization and a 8192-token limit. We first examined the role of task-specific instructions by replacing them with a generic prompt (Table S3) or removing instructions entirely, thereby isolating their contribution to the resulting embeddings. Next, we systematically excluded medical events from six different categories: demographics, visits, medications, procedures, labs, and conditions. To this end, we grouped medical codes into six mutually exclusive categories based on ontology prefixes and hierarchical SNOMED parent concepts (Table S8). Finally, we constructed serializations that retained only medical events from one category.

Effect of different time windows

To examine the influence of recency on predictive performance, we varied the time window preceding the prediction date used during EHR text serialization with a maximum context size of 8192 tokens. We evaluated Qwen3-Emb-8B, Qwen2-Emb-7B, Llama3.1-LLM2Vec-8B, and the count-based baseline with a GBM head across seven intervals: one hour, one day, one week (7 days), one month (30 days), one year (365 days), three years (1095 days), and full history. For each window, only events occurring within the specified interval before the prediction time were included. Thus, only data from the respective time window contributed to the aggregated information and visit data in the EHR serialization. All other aspects of the serialization, including structure, formatting, and instruction prompts, remained unchanged to isolate the effect of the temporal window.

Effect of different context sizes

We investigated the impact of varying context sizes in the LLM embedding models. Specifically, we evaluated Qwen3-Emb-8B, Qwen2-Emb-7B, and Llama3.1-LLM2Vec-8B with input token limits of 512, 1024, 2048, 4096, and 8192 tokens. Input tokens exceeding these thresholds were discarded. Due to the EHR list serialization, including the most recent occurrences of each medical code, additional input tokens primarily consisted of earlier medical concepts. All other preprocessing choices and task-specific instructions were held fixed across context-size settings. By testing these varying context sizes, we aimed to assess the balance between capturing historical medical data and preserving the clarity of high-priority information within the embeddings.