Model setting and data collection<\/p>\n
In this work, we utilized GPT-4 (version 20230613) as the backend model via the Azure OpenAI API, which is trained on data compiled before September 2021. The Azure OpenAI API is compliant with the Health Insurance Portability and Accountability Act, ensuring robust data privacy protection. To ensure stable and reproducible output, we set the temperature parameter to the absolute 0. The evaluated gene sets (Table 1<\/a>) were derived from recent releases by Hu et al.16<\/a> and Joachimiak et al.22<\/a> after 2023. The gene set corresponding to each ground truth in different datasets was determined by aggregating the genes with which it was directly annotated with those of all its ontological descendants.<\/p>\n Furthermore, we collected 191 gene sets from PubMed articles that were published from November 2021 to December 2023 to evaluate the performance of different LLMs in various data sources (Supplementary Table 1<\/a>). These gene sets range in size from 3 to 408, with an average of 35.67. The ground truths of these gene sets are released along with articles. The average word count in ground truth is 3.267. For model assessments, we developed the evaluation pipeline using Python (version 3.11.0) alongside PyTorch (version 1.13.0). Other necessary Python packages are NumPy (version 1.26.3), Pandas (version 2.1.4) and Seaborn (version v0.13.2).<\/p>\n Overview of GeneAgent<\/p>\n GeneAgent is a language agent built upon GPT-4 to automatically interact with domain-specific databases to annotate functions for gene sets, which is composed of four key steps: generation, self-verification, modification and summarization. Each module is triggered by a specific instruction tailored to its function (Supplementary Document 1<\/a>). The goal of GeneAgent is to generate a representative biological process name (P) for a set of genes, denoted as \\(D=\\{{{g}_{i}|}_{i=1}^{N}\\}\\). Each gene gi in this set is identified by its unique name, and the D is associated with a specific curated biological term, that is, ground truth (G). When provided with a D, GeneAgent outputs a \\(P\\), accompanied by analytical texts (A) detailing the functions of the genes involved, which can be formally defined as GeneAgent (D)\u2009=\u2009(P, A).<\/p>\n Pipeline of generating prominent biological process names<\/p>\n The gene set in the dataset D is separated by a comma (\u2018,\u2019) and serves as input parameters for the instruction of the generation (g) step. Following the generation stage, D is assigned with an initial process name \\(({P}_{\\rm{ini}})\\) and corresponding analytical narratives \\({(A}_{{ini}})\\), that is, \\({\\rm{GeneAgent}}_{g}(D)=({P}_{\\rm{ini}}\\,,\\,{A}_{\\rm{ini}})\\).<\/p>\n Afterwards, GeneAgent generates a list of claims for \\({P}_{{ini}}\\) by using statements like \u2018be involved in\u2019 and \u2018related to\u2019 to generate a hypothesis for the gene set and its process name. After that, GeneAgent activates selfVeri-Agent (Fig. 1c<\/a>) to verify each claim in the list. Initially, selfVeri-Agent extracts all gene symbols and the process name from the claims. Subsequently, it utilizes the gene symbols to invoke the appropriate APIs for the autonomous interaction with domain-specific databases, using their established knowledge to validate the accuracy of the process name. Finally, it assembles a verification report (\\({{\\mathcal{R}}}_{P}\\)) containing findings and decisions (that is, \u2018supported\u2019, \u2018partially supported\u2019 or \u2018refuted\u2019) to the input claim.<\/p>\n Next, GeneAgent initiates the modification (m) step to either revise or retain the \\({P}_{{ini}}\\) based on the findings in the \\({{\\mathcal{R}}}_{P}\\). If the \\({P}_{\\rm{ini}}\\) is determined to revise by GeneAgent, the \\({A}_{\\rm{ini}}\\) is also instructed to be modified accordingly, that is, \\({\\rm{GeneAgent}}_{m}\\left({P}_{\\rm{ini}},{{A}}_{\\rm{ini}},{{\\mathcal{R}}}_{P}\\right)=({P}_{\\mathrm{mod}},\\,{A}_{\\mathrm{mod}})\\). Following this, GeneAgent applies the self-verification to the \\({A}_{\\mathrm{mod}}\\) to verify the gene functions in the explanatory analyses while checking the updated process name again. This step is also started with generating a list of claims for different gene names and their function names and is finished with deriving a new verification report (\\({{\\mathcal{R}}}_{A}\\)) containing a decision of \u2018supported\u2019, \u2018partially supported\u2019 or \u2018refuted\u2019 made by the selfVeri-Agent.<\/p>\n Finally, based on the report \\({{\\mathcal{R}}}_{A}\\), both \\({P}_{\\mathrm{mod}}\\) and \\({A}_{\\mathrm{mod}}\\) are modified according to the summarization (s) instruction to generate the final biological process name (P) and the analytical narratives (A) of gene functions, that is, \\({\\rm{GeneAgent}}_{s}\\,({P}_{\\mathrm{mod}},{A}_{\\mathrm{mod}},\\,{{\\mathcal{R}}}_{A})=(P,A)\\).<\/p>\n Domain-specific databases configured in the selfVeri-Agent<\/p>\n In the self-verification stage, we configured four Web APIs to access 18 domain databases (Fig. 3c<\/a> and Supplementary Document 4<\/a>).<\/p>\n (1) g:Profiler27<\/a> (https:\/\/biit.cs.ut.ee\/gprofiler\/page\/apis\/<\/a>) is an open-source tool for GSEA. In GeneAgent, we used eight domain-specific databases: GO, KEGG41<\/a>, Reactome42<\/a>, WikiPathways43<\/a>, Transfac44<\/a>, miRTarBase45<\/a>, CORUM46<\/a> and Human Phenotype Ontology47<\/a> to perform enrichment analysis for the gene set. For each gene set, we used the g:GOSt interface to identify the top five enrichment terms along with their descriptions.<\/p>\n (2) Enrichr28<\/a>,29<\/a> (https:\/\/maayanlab.cloud\/Enrichr\/help#api\/<\/a>) is also a valuable tool for GSEA. We configured four databases related to the pathway analysis in the Enrichr API, that is, KEGG_2021_Human, Reactome_2022, BioPlanet_2019 (ref. 48<\/a>) and MSigDB_Hallmark_2020. In GeneAgent, we selected to return the top five standard pathway names via databases.<\/p>\n (3) E-utils30<\/a>,31<\/a> (https:\/\/www.ncbi.nlm.nih.gov\/<\/a>) is an API designed for accessing the NCBI databases for various biological data. In GeneAgent, we augment our repository of functional information associated with an individual gene by invoking its Gene database and PubMed database. Different databases can be used by defining the database parameter as Gene or PubMed in the foundation API.<\/p>\n (4) AgentAPI is our custom API library, developed using four gene-centric databases related to gene\u2013disease, gene\u2013domain, PPI and gene\u2013complex. GeneAgent calls the appropriate database by specifying the desired interface at the end of the basic API, and subsequently retrieving the top ten relevant IDs to gene functions. These IDs are then used to match their names in the corresponding database.<\/p>\n Notably, we implemented a masking strategy for APIs and databases during the self-verification stage to ensure unbiased assessments across various gene sets. Specifically, we removed the g:Profiler API when assessing gene sets collected from the GO dataset because it can perfectly derive their ground truths by accessing the GO database. Similarly, we masked the \u2018MSigDB_Hallmark_2020\u2019 database within the Enrichr API when evaluating gene sets collected from the MSigDB database.<\/p>\n Calculation of ROUGE score<\/p>\n Three distinct ROUGE metrics25<\/a> are used to access the recall of generated names relative to ground truths: that is, ROUGE-1 and ROUGE-2, which are based on n-gram, and ROUGE-L, which utilizes the longest common subsequence (LCS). The calculation formulas are as follows:<\/p>\n $${\\rm{ROUGE}}-{\\rm{N}}=\\frac{{\\sum }_{S\\in {\\rm{ref}}}{\\sum }_{{g}_{N}\\in S}{\\rm{count}}_{\\rm{match}}\\left({g}_{N}\\right)}{{\\sum }_{S\\in {\\rm{ref}}}{\\sum }_{{g}_{N}\\in S}{\\rm{count}}\\left({g}_{N}\\right)},\\text{N}=1,\\,2$$<\/p>\n $$\\left\\{\\begin{array}{c}{R}_{\\rm{lcs}}=\\frac{\\rm{LCS}\\left({\\rm{ref}},{\\rm{hyp}}\\right)}{m}\\\\ {P}_{\\rm{lcs}}=\\frac{{\\rm{LCS}}\\left({\\rm{ref}},{\\rm{hyp}}\\right)}{n}\\\\ {\\text{ROUGE}}-{\\text{L}}\\,=\\frac{\\left(1+{\\beta }^{2}\\right){R}_{\\rm{lcs}}{P}_{\\rm{lcs}}}{{R}_{\\rm{lcs}}+{\\beta }^{2}{P}_{\\rm{lcs}}}\\end{array}\\right.$$<\/p>\n Here, the \u2018ref\u2019 denotes the reference terms and \u2018hyp\u2019 denotes the generated names. m and n are the token lengths of \u2018ref\u2019 and \u2018hyp\u2019, respectively. \u03b2 is a hyper-parameter.<\/p>\n Calculation of semantic similarity<\/p>\n After generating the biological process name (P) for the gene set D, the semantic similarity between \\(P\\) and its ground truth (G) is computed by MedCPT26<\/a>, a state-of-the-art model for language representation in the biomedical domain. It is built based on PubMedBERT49<\/a> with further training using 255 million query\u2013article pairs from PubMed search logs. Compared with SapBERT50<\/a> and BioBERT51<\/a>, MedCPT has higher performance in encoding the semantics of biomedical texts.<\/p>\n a.<\/p>\n Calculation of semantic similarity between P and G<\/p>\n First, P and G are encoded by MedCPT into embeddings, and then the cosine similarity between their embeddings is calculated, yielding a score in the interval [\u22121, 1]. Finally, we take the average value of all similarity scores to evaluate the performance of GeneAgent on gene sets in each dataset.<\/p>\n b.<\/p>\n Calculation of background semantic similarity distribution<\/p>\n First, P is paired with all possible terms \\({G}_{i}{\\mathcal{\\in}}{\\mathcal{Q}}\\), where Q denotes 12,320 candidate terms consisting of 12,214 GO biological process terms, and all available terms in NeST (50) and MSigDB (56). Then, P and Gi are fed into MedCPT to get the embeddings, that is, \\(\\mathop{P}\\limits^{ \\rightharpoonup }\\) and \\(\\mathop{{G}_{i}}\\limits^{ \\rightharpoonup }\\). Afterwards, we calculated the cosine similarity for each \\(< \\mathop{P}\\limits^{ \\rightharpoonup },\\mathop{{G}_{i}}\\limits^{ \\rightharpoonup } >\\) pair. Finally, we ranked all cosine scores from large to small and observed the position where the pair <P,Gp> (Gp is the ground truth for P) located in. The higher position denotes the generated names have a higher similarity score to their ground truths than other candidate terms.<\/p>\n Calculation of hierarchical semantic similarity<\/p>\n We first collected the hierarchical structures of all GO terms from the GoBasic.obo file in the GO database (2023-11-15 version), yielding 1,951,615 GO term pairs across five relationships: \u2018is a\u2019, \u2018part of\u2019, \u2018regulates\u2019, \u2018negatively regulates\u2019 and \u2018positively regulates\u2019. Next, we extracted the ancestral terms for all ground truths in the evaluation datasets used in this study, limiting the distance from each ground truth to its respective ancestor to within three hops. Finally, we calculated the semantic similarity between each ancestral term and its corresponding ground truth to assess whether the generated names achieved a higher similarity score with ancestral terms. For our evaluation, we computed hierarchical semantic similarity exclusively for gene sets in the GO dataset.<\/p>\n Pipeline of enrichment term test using verification reports<\/p>\n For gene sets in MSigDB, we first collected its verification report produced by the selfVeri-Agent of GeneAgent. Afterwards, each gene set and the associated report were used as the parameters of the instruction (Supplementary Document 1<\/a>) for the GPT-4. Therefore, GPT-4 can summarize multiple enrichment terms for the given gene set. Finally, we used the exact match to evaluate the accuracy of the tested terms summarized by the GPT-4. Specifically, for each gene set in the evaluation, we first utilized g:Profiler to perform GSEA, where the P-value threshold is set to 0.05. Then, we obtained significant enrichment terms for the given gene sets as the ground truth. Finally, we counted the number of tested terms summarized by GPT-4 that correctly match the significant enrichment term of each gene set. One tested term is considered as accurate only when there is an exact match between all the words in the tested term and one term in the ground truth.<\/p>\n Human checking for the decisions of selfVeri-Agent<\/p>\n We randomly selected ten gene sets from NeST with 132 claims for human inspection. There are two parts in the verification report: the claims and the decisions to the claims along with evidence (Supplementary Document 2<\/a>). Annotators were asked to label the selfVeri-Agent decisions (that is, support, partially support and refute) for each claim and judge whether such decisions are correct, partially correct or incorrect, which follows the study of natural language inference52<\/a> and fact verification53<\/a>. For each claim, the annotators need to make a judgment based on assertions of the gene (set) functions provided in the evidence:<\/p>\n a.<\/p>\n Correct: This category applies when GeneAgent\u2019s decision completely aligns with the evidence supporting the input claim. The decision is considered correct if it accurately reflects the evidence documented, demonstrating a clear and direct connection between the claim and the supporting data.<\/p>\n b.<\/p>\n Partially correct: It is designated when GeneAgent\u2019s decision requires indirect reasoning or when the decision, although related, does not completely align with the direct evidence provided. This occurs when the decision is somewhat supported by the evidence but requires additional inference or context to be fully understood as supporting the input claim.<\/p>\n c.<\/p>\n Incorrect: This category is used when GeneAgent\u2019s decision either contradicts the evidence or lacks any substantiation from the verification report.<\/p>\n Melanoma gene sets in the preclinical study<\/p>\n The mouse B2905 melanoma cell line, which is derived from a tumor from the M4 model, where melanoma is induced by ultraviolet irradiation on pups of hepatocyte growth factor-transgenic C57BL\/6 mice54<\/a>.<\/p>\n Specifically, 24 single cells were isolated from the parental B2905 melanoma line and then expanded to become individual clonal sublines (that is, C1 to C24)55<\/a>. Each of these 24 sublines was subjected to whole-exome sequencing and full-transcript single-cell RNA sequencing using the Smart-seq2 protocol. The single nucleotide variants called from exome sequencing results were used to build the tumor progression tree for all the 24 sublines. Based on the in vivo growth and therapeutic responses of the sublines in the clusters, three clades are named as \u2018high aggressiveness and resistant (HA-R)\u2019, \u2018high aggressiveness and sensitive (HA-S)\u2019 and \u2018low aggressiveness and sensitive (LA-S)\u201932<\/a>. Afterwards, EvoGeneX56<\/a> is applied to the single-cell RNA-sequencing data of the 24 clonal sublines, where the phylogenetic relation is defined by the mutation-based tumor progression tree, to identify adaptively upregulated and downregulated genes in each of HA-R, HA-S and LA-S clades. The adaptively upregulated and downregulated gene lists were then subjected to the Kyoto Encyclopedia of Genes and Genomes pathway enrichment analysis. The genes in the enrichments and their enriched terms are used to test the GeneAgent. In our case study, we only utilized the seven gene sets analyzed from the clonal subline as the evaluation data of GeneAgent. We did not access or process any original data from clinical experiments.<\/p>\n Human annotation for outputs in the case study<\/p>\n For the assessment of different outputs between GeneAgent and GPT-4, we established four criteria following the existing studies on the evaluation of LLMs57<\/a>,58<\/a>.<\/p>\n a.<\/p>\n Relevance: Assess whether the content about genes pertinently reflects their functions, providing value to biologists.<\/p>\n b.<\/p>\n Readability: Evaluate the fluency and clarity of the writing, ensuring it is easily understandable.<\/p>\n c.<\/p>\n Consistency: Determine whether the analytical narratives align consistently with the specified process name.<\/p>\n d.<\/p>\n Comprehensiveness: Verify whether the outputs provide a comprehensive understanding of gene functions.<\/p>\n Based on these four established criteria, two experts are tasked with evaluating the final responses from the outputs of GPT-4 and GeneAgent. They operate the annotation under a blind assessment protocol, where they are unaware of the algorithm that produced each response. Their main responsibility is to annotate and compare the preference for outputs generated by GPT-4 versus GeneAgent. They carefully review and select the more effective response, justifying their selections with relevant comments. Following a comprehensive synthesis of all feedback, these two experts are required to make a definitive judgment on which output most effectively satisfies the users\u2019 requirement (Supplementary Document 3<\/a>).<\/p>\n Reporting summary<\/p>\n