OpenScholar<\/p>\n
OpenScholar (detailed in Extended Data Fig.\u20091<\/a>) is a new retrieval-augmented LM designed to ensure reliable, high-quality responses to a range of information-seeking queries about scientific literature.<\/p>\n Task formulation and challenges<\/p>\n Given a scientific query x, the task is to identify relevant papers, synthesize their findings and generate a response y that effectively addresses the query. This response should be accompanied by a set of citations, C\u2009=\u2009c1,\u2009c2,\u2026,\u2009cK, in which each citation ci corresponds to an existing scientific paper. Each ci in C corresponds to specific passages from scientific literature and should be provided as an in-line citation, linked to the relevant spans of text in y, following standard practice in scientific writing. These citations allow researchers to trace the output back to the original literature, ensuring transparency and verifiability.<\/p>\n However, this task presents several challenges: (1) retrieving high-recall, high-precision scientific content from a vast, domain-specific corpus; (2) synthesizing accurate, non-hallucinated responses grounded in the retrieved evidence; and (3) producing citation-aware outputs that align generated text with appropriate references at a fine-grained level. A further challenge lies in the scarcity of resources: to our knowledge, there is limited availability of large-scale, up-to-date scientific corpora, especially those suitable for dense retrieval, as well as a lack of supervised training data for both retrieval and generation in scientific domains.<\/p>\n Overview of OpenScholar<\/p>\n To address these challenges, OpenScholar introduces several key innovations that extend the standard RAG (refs.\u20091<\/a>,5<\/a>) model for scientific literature synthesis. Specifically, OpenScholar combines domain-specialized retrieval, citation-aware generation and a new self-feedback inference mechanism, all built on top of a fully open and large-scale scientific data store.<\/p>\n Formally, OpenScholar consists of three key components: a data store D, a retriever \\({R}\\) and a generator LM \\({G}\\). In standard retrieval-augmented inference pipelines, the process begins with \\({R}\\), which retrieves a set of passages P\u2009=\u2009{p1,\u2009p2,\u2026,\u2009pN} from D\u2014a large-scale corpus of previously published scientific papers\u2014based on semantic relevance to the input query x. These passages serve as context for the next step. The generator LM \\({G}\\) then takes both the retrieved passages P and the input query x to produce the output y along with corresponding citations C. Formally, this process can be represented as: <\/p>\n $$y,{\\bf{C}}={G}(x,{R}(x,{\\bf{D}})),$$<\/p>\n in which each ci in C corresponds to a specific passage from P.<\/p>\n OpenScholar introduces new technical contributions to address the aforementioned challenges. (1) To address the lack of large-scale, up-to-date scientific corpora, we construct OSDS, a database of 45\u2009million scientific papers with precomputed dense embeddings, representing, to our knowledge, the largest and most up to date scientific paper data store available. (2) To enable high-recall, high-precision retrieval and support LM training in scientific domains, we design a retrieval pipeline that integrates both our trained OpenScholar retriever and OpenScholar reranker, optimized on scientific data to select the top N passages for the generator \\({G}\\)\u2014and complementary retrieval APIs\u2014ensuring broader coverage and improved relevance. (3) To improve factuality and evidence grounding, we introduce iterative self-feedback inference with retrieval and citation verification, in which the LM first produces an initial draft y0 with \\({G}\\) and then iteratively refines it using retrieval-augmented self-feedback. (4) To enhance citation accuracy and overall output quality, we use this inference pipeline to generate high-quality training data, enabling the training of specialized LMs that produce more accurate and citation-aware long-form answers.<\/p>\n OpenScholar retrieval pipeline<\/p>\n Extended Data Fig.\u20091<\/a> (top left) shows our retrieval pipeline, consisting of a data store D, a bi-encoder retriever \u03b8bi and a cross-encoder reranker \u03b8cross. We first select initial candidate paragraphs using D and \u03b8bi, as well as external APIs, and then refine and identify the top N relevant paragraphs using \u03b8cross.<\/p>\n Scientific paper collection and data store construction<\/p>\n Although previous work often used a small subset of scientific papers, such as arXiv papers from 2023 to 2024 (ref.\u20099<\/a>), it is important to have a diverse set of papers to improve the quality and coverage of model generation23<\/a>. For this, we use peS2o (ref.\u200924<\/a>) as our retrieval source, which consists of open-access academic papers from S2ORC (ref.\u200925<\/a>). We built our data store using peS2o\u2009v3, which includes 45\u2009million papers up to October 2024. For evaluations, we use peS2o\u2009v2, which consists of papers up to January 2023, as our main benchmarks and models were constructed before the curation of peS2o\u2009v3. Our data store, which we call OSDS, consists of 236\u2009million passages. To our knowledge, this is the largest open-sourced data store for scientific literature.<\/p>\n Initial paragraph retrieval<\/p>\n We retrieve passages from three sources: (1) the OSDS using our trained retriever; (2) publicly available abstracts from papers returned through the Semantic Scholar API (ref.\u200926<\/a>) based on search keywords; and (3) publicly available texts from papers retrieved through a web search engine using the original query x.<\/p>\n For (1), we first generate embeddings of each passage in the OSDS D using the passage bi-encoder \u03b8bi, which processes text chunks (for example, queries or passages) into dense vectors27<\/a> offline. Off-the-shelf retrieval models often struggle in out-of-domain scenarios28<\/a>. To overcome this limitation, we develop \u03b8bi by continually pre-training Contriever29<\/a> on the peS2o data store in an unsupervised fashion to improve domain-specific retrieval performance. During inference, we encode the query using \u03b8bi and retrieve the top 70 passages through a nearest-neighbour search27<\/a>. Following previous work23<\/a>, we split the main text of each paper into discrete, 256-word text blocks (as determined by white space) and concatenate the paper title to each block to formulate passages in D. Although semantic segmentation can be used to split scientific articles into meaningful sections, we found that not all papers in our data store consistently retain such semantic or discourse structures. Furthermore, applying segmentation models post hoc would be computationally expensive at this scale. Therefore, following common practice in this area27<\/a>,29<\/a>, we divide articles into fixed-length chunks to ensure scalability and simplicity. Therefore, several text chunks from the same paper can be retrieved at inference time.<\/p>\n For (2), we first generate keywords from the query x using a generator LM. These keywords are then used to retrieve the top 10 papers for each, as ranked by citation count, through the Semantic Scholar search API. This approach addresses a limitation of the Semantic Scholar API, which cannot effectively handle long, question-like search queries. If the full text is available in HTML format (for example, ar5iv), we retrieve the entire text and include all passages from the paper as candidate documents. Otherwise, we only consider the abstract.<\/p>\n For (3), we obtain the top 10 search results using the You.com retrieval API, restricting the search to academic platforms such as arXiv and PubMed. Similarly to (2), if the papers are open access, we extract and add their full texts to the candidate pool; otherwise, we include only their abstracts.<\/p>\n Top N paragraph reranking and finalization<\/p>\n After the initial stage, we have gathered more than a hundred or even a thousand relevant passages per query. However, passages retrieved by the bi-encoder may include unhelpful context owing to deep interactions between a query and passages, as they are encoded separately30<\/a>. Feeding a large number of documents that might include irrelevant content to LLMs can cause efficiency and performance issues, even with state-of-the-art models31<\/a>,32<\/a>. To overcome these challenges, we use a cross-encoder reranker33<\/a>,34<\/a>, denoted as \u03b8cross. For each candidate paragraph, the cross-encoder reranker jointly encodes and computes the relevance score between the input query and each of the passages. We then use the relevance score to rank the passages accordingly. To train \u03b8cross for scientific domains, we fine-tune a BGE reranker34<\/a> using synthetic data generated by Llama-3-70B-Instruct. Specifically, we randomly generate queries based on abstracts from peS2o and retrieve the top 10 passages. For each passage, Llama-3-70B-Instruct assigns a relevance score from 1 to 5, for which we consider scores of 4 or 5 as positive and scores of 1 or 2 as negative. Passages with a score of 3 are discarded. More details of \u03b8cross training are in Supplementary Information Section\u20093.3<\/a>. During reranking and finalization of the top N passages, we also implement extra meta-filtering, which includes: (1) limiting the number of passages per paper to three passages and (2) incorporating normalized citation counts into relevance scores predicted by the cross-encoder.<\/p>\n Inference: self-reflective iterative RAG<\/p>\n In standard RAG (refs.\u20095<\/a>,35<\/a>), a generator LM takes in the original input x and top N retrieved passages P and generates the output y0. Although effective for tasks such as question answering2<\/a>, this one-step generation can lead to unsupported claims36<\/a> or incomplete output owing to missing information7<\/a>,37<\/a>. To address these challenges, in OpenScholar, we introduce an iterative generation approach with self-feedback, which involves three steps: (1) initial response and feedback generation to output the initial draft y0 and a set of feedback on y0; (2) iterative refinement with further retrieval to improve y0 using the feedback; and (3) citation verification. Our inference is detailed in Extended Data Fig.\u20091<\/a>, top\u00a0right.<\/p>\n Initial response and feedback generation<\/p>\n Given the input x and retrieved passages P, the generator LM first produces an initial response y0 with citation markers tied to the corresponding passages in P. After generating y0, the LM generates a set of feedback on y0, F\u2009=\u2009f1,\u2009f2,\u2026,\u2009fT, that is aimed at improving the initial response, in which each feedback ft is a natural language sentence that describes potential improvements. Although the model can generate an arbitrary number of feedback (T), we set a maximum limit of three feedback sentences for efficient inference. Unlike previous work that relies on a predefined set of feedback signals7<\/a>, our approach allows the LM to generate flexible natural language feedback on various aspects of the response, such as organization, completeness or further required information. If the feedback sequence identifies missing content (for example, \u201cThe answer only includes empirical results on QA tasks. Add results from other task types.\u201d), the LM also generates a retrieval query for further retrieval using the pipeline.<\/p>\n Iterative refinement<\/p>\n We then iterate over the feedback F to incrementally refine the output. If fk indicates that further retrieval is needed, the query qk is used to retrieve extra passages, which are appended to P before producing yk. Although we could iteratively regenerate the output each time feedback is provided, doing so introduces more latency. Empirically, we found that feedback is often diverse, addressing different aspects of generation. As a result, sequentially incorporating feedback from the initial output remains effective. The LM uses the previous output yk\u22121, the retrieved passages P and newly retrieved passages, if any, to generate a revised output yk. This process is repeated until all feedback has been addressed, resulting in a final output yT by time step T.<\/p>\n Citation verification<\/p>\n Finally, we instruct the generator LM to verify the citations in yt. Specifically, the generator ensures that all citation-worthy statements\u2014scientific claims requiring justification\u2014are adequately supported by references from the retrieved passages. If any claims lack proper citations, the LM performs a post-hoc insertion to ensure that citation-worthy statements are supported by passages. In our pipeline, we do not remove sentences that lack citation-worthy statements.<\/p>\n Synthetic\u00a0training data generation with inference pipeline<\/p>\n Building powerful LMs that can effectively synthesize scientific literature is challenging because of the lack of training data for this problem. Although there are some resources to train scientific LMs38<\/a>, most tasks do not require open-retrieval settings and are single-paper tasks. As a result, most previous work in this area10<\/a> relies on proprietary LMs, which poses challenges for reproducibility and inference costs.<\/p>\n We use our inference-time pipeline to synthetically generate high-quality training data through self-feedback, so that the resulting model can get better at generating higher-quality output without going through the self-feedback process (Extended Data Fig.\u20091<\/a>, bottom).<\/p>\n Question and response generations<\/p>\n Our data generation process involves three steps: first, selecting the top-cited papers from D; second, generating information-seeking queries based on their abstracts; and third, using the OpenScholar inference-time pipeline to produce high-quality responses. We generate data using Llama\u20093.1\u200970B (ref.\u200917<\/a>). Specifically, we begin by sampling 1\u2009million paper abstracts from the peS2o dataset and gathering their corresponding metadata, such as publication year or citation count. We then randomly select 10,000\u2009papers that were published after 2017 and prompt a LM to generate literature review questions or information-seeking queries based on each abstract that require several papers to answer. Next, we use our OpenScholar pipeline to produce the final output yT, along with intermediate generations such as feedback F and initial outputs.<\/p>\n Data filtering<\/p>\n Despite its effectiveness and scalability, synthetic data may also contain issues such as hallucinations, repetitive writing or limited instruction-following39<\/a>. To address this, we introduce a two-step data filtering process: pairwise filtering and rubric filtering, using the same LM as for data generation. In pairwise filtering, we compare the quality of model outputs yT (output at the final step) and y0 (initial output) and retain the output that is judged to be higher quality. We find that y0 is preferred over yT around 20% of the time, owing to over-editing or increased redundancy after several iteration steps. We then evaluate the quality of the chosen response on a five-point scale across two aspects: organization and factual precision and citation accuracy. A valid model output must achieve a score of 4.5 or higher in both categories and we discard instances whose outputs do not meet this requirement.<\/p>\n Data mixing and training<\/p>\n From this synthetic pipeline, we generate three types of training data: answer generation (x\u2009\u2192\u2009y), feedback generation (y0\u2009\u2192\u2009F) and feedback incorporation (yt\u22121,\u2009ft\u2009\u2192\u2009yt). We found that incorporating both final and intermediate outputs during training helps smaller LMs learn to generate more effective feedback. We further blend this synthetic training data with existing general-domain instruction-tuning data40<\/a> and scientific instruction-tuning data38<\/a>, ensuring that 50% of the training data come from scientific domains, whereas the remaining 50% is sourced from general-domain data. We also generate synthetic fact verification and Boolean QA data based on sampled abstract data from peS2o. For this, we sort the papers based on citation count and select the top 100,000\u2009papers. After data mixing, we train generator LMs on our large-scale synthetic training data. We train Llama-3.1-8B-Instruct on the generated training data.<\/p>\n OpenScholar experimental details<\/p>\n We use peS2o\u2009v2 as D, our default data store. For \u03b8bi and \u03b8cross in OpenScholar, we use our trained bi-encoder and cross-encoder models, which consist of 110\u2009million and 340\u2009million parameters, respectively. We analysed various cross-encoder and bi-encoder models on a customized synthetic benchmark and found that OpenScholar retriever (bi-encoder) and OpenScholar reranker (cross-encoder) achieved the highest normalized discounted cumulative gain among models of comparable size (Supplementary Information Section\u20095.2<\/a>. We set the maximum number of papers from web search and Semantic Scholar to 10. For the generator LMs, we set the temperature to 0.7 and limit the maximum token count to 3,000 for response generation and 1,000 for feedback generation and use the vLLM package for faster inference. We trained Llama\u20093.1\u20098B for two epochs on 130,000\u2009training instances for two epochs. For all models, we set the number of passages input into the generator LM to five for single-paper tasks and ten for multi-paper tasks. No few-shot demonstrations are provided, except for SciFact and PubMed, for which we include one-shot demonstrations. OpenScholar responses are marked with special decorators Response_Start and Response_End and citations are indicated as reference numbers (for example, [1]), which correspond to the reference documents provided in the context. We do not add any new special tokens to the model vocabulary; instead, we use these decorators as regular strings. After training, we observe that the model can generate the correct tokens as intended.<\/p>\n ScholarQABenchChallenges and overview<\/p>\n Previous studies on building LMs to synthesize scientific literature use either small-scale, single-domain human evaluation8<\/a>,9<\/a> or oversimplified multiple-choice QA set-ups10<\/a>. Building high-quality benchmarks for literature review has two main challenges. First, creating such datasets is resource-intensive, as it requires PhD-level domain expertise and research experience, particularly when annotating realistic questions and high-quality answers. Second, even when high-quality data are available, reliably evaluating long-form natural language responses presents a notable challenge, especially in expert domains13<\/a>,14<\/a>. This contrasts with benchmarks for other scientific processes, such as automated experimental code generation, for which clearer evaluation criteria, such as pass@1, are more readily available45<\/a>.<\/p>\n To address these gaps, we introduce ScholarQABench, a benchmark that supports diverse formats of scientific literature synthesis tasks, including closed-form classification, multiple-choice and long-form generation, as shown in Extended Data Table\u20091<\/a>. We use three existing single-paper datasets and then construct a suite of high-quality, expert-annotated datasets for computer science, biomedicine, physics and neuroscience. We also built a reliable automatic evaluation pipeline. Extended Data Fig.\u20092<\/a> shows an example and an overview of the evaluation pipeline.<\/p>\n Data curation<\/p>\n ScholarQABench is designed to evaluate model capabilities in automating scientific literature review. The curation process is guided by three key factors. Diversity of tasks: ScholarQABench includes tasks with a range of input-output formats. Diversity of disciplines: unlike previous analyses that often focus on a single discipline such as computer science, ScholarQABench spans four scientific disciplines. Inclusion of multi-paper tasks: unlike previous work that focuses on understanding single, preselected papers, all tasks require retrieving from the entire open-access collection of full texts of papers and four datasets specifically require reasoning over several retrieved papers. As a result, ScholarQABench is the first multidisciplinary literature synthesis benchmark that requires long-form generation grounded in several recent papers, with all examples annotated by PhD-level experts. This sets it apart from previous datasets that focus on short-form or multiple-choice answers or rely on static scientific knowledge reasoning10<\/a>,11<\/a>,12<\/a>,46<\/a>, as well as those that lack expert-annotated refs. 13<\/a>,47<\/a>.<\/p>\n Note that our benchmark is designed for single-turn set-ups and does not include multi-turn follow-up questions and answers in dynamic evaluations48<\/a>. Evaluating multi-turn LM\u2013human interactions remains challenging49<\/a>, so we begin with a single-turn, static evaluation set-up as a first step towards more realistic assessments of such systems.<\/p>\n Single-paper tasksSciFact<\/p>\n SciFact42<\/a> is a dataset of 1,400\u2009expert-written scientific claims in the biomedical domain, paired with gold evidence from existing PubMed paper abstracts annotated with labels and rationales. We include validation set queries labelled as either \u2018supports\u2019 (true) or \u2018contradicts\u2019 (false), discarding the original gold evidence, and reformulate the task as binary open retrieval, in which a system needs to identify relevant papers from a large collection of papers.<\/p>\n PubMedQA<\/p>\n PubMedQA41<\/a> has expert-annotated (yes\/no\/maybe) QA data on PubMed paper abstracts. Similarly to SciFact, we only keep instances with yes or no labels and discard the original abstract passage to formulate the task as an open-retrieval set-up.<\/p>\n QASA<\/p>\n QASA43<\/a> is a single-paper QA dataset that consists of question answering pairs, requiring reasoning over scientific articles in artificial intelligence and machine learning. We evaluate the ability of the model to sufficiently answer a detailed question about the target paper. Although the original dataset provides three subtasks (answer selection, rationale generation and answer compositions) as well as end-to-end QA, we evaluate the performance of the models based on an end-to-end QA set-up.<\/p>\n Multi-paper tasks<\/p>\n Single-paper, closed-set tasks may provide reliable evaluations. However, they may not be reflective of realistic scenarios, in which complex, open-ended questions are asked independently from existing papers and require multi-paper retrieval and reasoning. Few datasets13<\/a>,47<\/a> explore multi-paper set-ups with realistic queries and most lack a reliable evaluation pipeline or human-written references. We address this gap by recruiting expert-level annotators across several scientific disciplines and curating three new long-form QA datasets for this challenging setting. All answers are written by PhD-level experts, with each taking approximately one hour to compose, reflecting the demanding nature of the task. Details of our annotation process, including compensation (US$30\u201345 per hour on average), are provided in Supplementary Information Section\u20092.3<\/a>. The process was approved by the ethics board (institutional review board) as exempt research. Data collection took place between April and October 2024 and all reference answers (where applicable) are grounded in scientific literature published up to October 2024. Below, we discuss each subset of the four multi-paper tasks, which span four broad scientific disciplines.<\/p>\n Scholar-CS<\/p>\n We collected 100\u2009questions along with detailed answer rubrics for each question across various computer science disciplines by recruiting expert annotators holding PhDs in the field (professors, postdoctoral researchers and research scientists). Annotators were tasked with writing literature review questions that require several research papers to answer. The question topics span areas such as networks, algorithms, the Internet of things, artificial intelligence and human\u2013computer interaction. Then, for each question, two other annotators searched the web to produce a rubric listing the key ingredients for a correct answer, categorized by importance (\u2018must have\u2019 and \u2018nice to have\u2019), along with supporting quotes from sources for each ingredient. The annotators were instructed not to use any LLM services for this initial part of the task. After the initial web search, the annotators were shown corresponding responses from four LLM services (Claude 3.5 Sonnet, GPT-4o, Perplexity Pro and an unpublished RAG prototype based on Claude 3.5) in a randomized order in case they wanted to revise their rubrics. On average, each question is annotated with 4.4 key ingredients, each supported by 4.4 quotes. Furthermore, we collected 31 expert-written long-form answers, authored by a separate pool of PhD-level annotators, to serve as a measure of expert human performance.<\/p>\n To measure agreement, we had both annotators produce rubrics for a subset of ten randomly sampled questions. We then compute the scores for responses from the four LLM services to which the annotators were exposed using our automated approach, once for each set of annotator rubrics. Finally, we calculate Pearson\u2019s correlation coefficient among the scores for each question and compute the average. Given the subjectivity of rubric annotation, we assess agreement both with and without the general criterion included in the scores, resulting in values of 79.3 and 59.5, respectively. Extended Data Fig.\u20091<\/a> shows an example.<\/p>\n Scholar-Bio and Scholar-Neuro<\/p>\n We further collected 2,759\u2009expert-written literature review questions in biomedicine and neuroscience, recruiting six experts who have a PhD in relevant areas and are at present research scientists and engineers. The annotators were asked to choose papers from their area of expertise and generate complex scientific questions that biomedical scientists might reasonably ask about the scientific literature based on their parsing of those papers. We collected questions from different areas, such as bioimaging, genetics, microbiology and neuromodulation, for each. Owing to the cost of annotation, we focused only on curating the questions.<\/p>\n Scholar-Multi<\/p>\n Last, we collected 108 literature review questions and expert-written answers with citations in three domains: computer science (artificial intelligence\/machine learning, human\u2013computer interaction), biomedicine (bioimaging, genetics) and physics (astrophysics, photonics, biophysics). All annotations are conducted by PhD students or postdoctoral scientists who have more than three years of research experience in the corresponding areas and have several first-author publications. We asked them to come up with questions that are related to the most recent literature and to compose answers to the questions using relevant papers that they found by means of a search. Our annotators were instructed not to use any LLM-based systems such as ChatGPT and told to only use general search (for example, Google Search) or paper search (for example, Semantic Scholar) systems. Statistics of collected questions are available in Table\u20093<\/a>, The distribution of subjects is shown in Supplementary Information Fig.\u20091<\/a>, along with the average annotation time per subject. We show several examples in Supplementary Information Figs.\u200912<\/a>\u201315<\/a>. On average, each annotator spent 56\u2009minutes per instance.<\/p>\n Metrics and evaluation protocols<\/p>\n We developed a multifaceted automatic evaluation pipeline to facilitate reproducible and efficient evaluations, complementing expert assessments. An overview of our evaluations is in Extended Data Fig.\u20092<\/a>.<\/p>\n Correctness<\/p>\n Correctness evaluates the degree of overlap or agreement between model-generated answers and human-annotated reference answers. This metric is applied only to tasks for which reference answers are available. For single-paper tasks, we directly compare the model outputs to gold reference texts, following the evaluation methodologies proposed in previous work41<\/a>,42<\/a>,43<\/a>. We refer to this metric as accuracy for simplicity. For SciFact and PubMedQA, which have fixed answer classes, we use exact match as the correctness metric. For QASA, we use ROUGE-L as an evaluation metric, following ref.\u200943<\/a>.<\/p>\n However, such approaches that rely on a single reference answer often fail to capture all valid outputs, especially in tasks requiring long-form answers synthesized from several papers, such as our multi-paper tasks. To address this, we introduce a new correctness evaluation framework based on Scholar-CS\u2019s expert-annotated rubrics, which we refer to as rubric score (rubric-based evaluation). Specifically, we combine two components: annotation-driven criteria (60%), which assess the presence of key content elements (\u2018ingredients\u2019) identified by annotators as necessary for a good answer, and general criteria (40% of the score), which evaluate aspects such as length, domain expertise, citation quality and use of supporting excerpts. GPT-4o Turbo scores each criterion and we compute a weighted sum to obtain the final correctness score. We conducted expert evaluations to measure the agreement between human and LLM judges on whether a rubric item was satisfied by a LLM-generated answer, using outputs from two LM systems and two expert annotators. The average agreement between the two human annotators was 0.80, whereas the average agreement between a human annotator and the LLM judge was 0.79. We conducted an analysis on the agreement between an evaluator, LM and a human, and the average correlation between humans was 0.62 and the average correlation between humans and the LLM judge was 0.81. More details are in Supplementary Information Section\u20092.3.1<\/a>.<\/p>\n Citation accuracy<\/p>\n Evaluating long-form responses to literature review questions requires citation accuracy: LMs should correctly attribute relevant evidence for all citation-worthy statements. In ScholarQABench, all systems generate outputs with reference numbers (for example, [1], [2]) linked to passages provided during inference. Following previous work36<\/a>,50<\/a>, we check whether each citation-worthy statement has appropriate citations and whether the citations support the statement (citation recall). For each citation, we then verify its relevance and necessity\u2014specifically, whether the citation supports the statement and whether its removal affects the integrity of remaining citations (citation precision). Finally, we compute citation F1 and use it as a primary metric for citation accuracy. Citation accuracy does not require gold reference answers or rubrics, so we apply this evaluation across all tasks.\u00a0More details are in Supplementary Information Section 2.3.3<\/a>.\u00a0<\/p>\n Content quality and organization on Scholar-Multi<\/p>\n We extend our evaluation beyond correctness and citation accuracy by defining further key aspects: relevance to the question, coverage in terms of topic breadth (for example, diversity of discussed papers) and depth (for example, sufficiency of details) and organization and writing flow. These aspects are difficult to capture using standard automatic metrics. We developed detailed instructions and five-point rubrics for each aspect and applied the same rubrics to both LLM and expert human evaluations. For the LLM judge, we use Prometheus v2 (ref.\u200944<\/a>), a state-of-the-art open-source model for fine-grained evaluation, chosen to ensure reproducibility and avoid the instability and cost issues associated with proprietary models51<\/a>. For human evaluations, when conducted by expert annotators on those three aspects, we also assess overall usefulness (usefulness). As previous studies show that LLM judges are less reliable when gold reference answers are not available52<\/a>, this evaluation is only applied to a task with human-annotated reference answers, namely Scholar-Multi. We analysed the agreement between human and model assessments on fine-grained aspects. We found that, although the model and humans sometimes disagreed on adjacent categories\u2014particularly between scores of 4 and 5\u2014the evaluations of the model aligned well with human rankings and its accuracy on a collapsed three-point rating exceeded 80% across different aspects and subject LMs.\u00a0More details are in Supplementary Information Section 2.3.2<\/a>.<\/p>\n Related workScientific LMs<\/p>\n Scientific LMs have spanned various domains, including biomedical53<\/a>,54<\/a>,55<\/a>, medical56<\/a>,57<\/a>,58<\/a>,59<\/a>, biomedical60<\/a>,61<\/a>,62<\/a>, geoscience63<\/a> and astronomy64<\/a>, with some models such as SciGLM65<\/a> and Uni-SMART66<\/a> that aim to cover diverse scientific domains in a single model. Recently, several works show that powerful general-purpose LLMs can also show strong capabilities in scientific tasks, such as medical question answering56<\/a>,67<\/a>, chemistry experimentation68<\/a> and applied mechanics69<\/a>. However, the reliance of a LM on information memorized within its parameters leads to frequent hallucinations in its output70<\/a>.<\/p>\n LMs to assist scientists<\/p>\n Recent studies have also examined the capabilities of LLMs to assist scientists in performing a range of scientific procedures, including generating new research ideas71<\/a>,72<\/a> and automating experimental code generation73<\/a>,74<\/a>. Our work, however, focuses specifically on benchmarking and developing methods for automating literature reviews and addressing questions related to up-to-date research\u2014tasks that are crucial to, and particularly challenging for, scientific inquiry. Several concurrent studies have attempted to build retrieval-augmented pipelines using proprietary LLMs and external APIs (for example, the Semantic Scholar API) for scientific literature review agents8<\/a>,10<\/a>,75<\/a>. Although these studies and our research all explore the potential of retrieval-augmented LMs in automating literature synthesis, previous works often relied on proprietary, black-box systems and limited evaluations, which commonly entail small-scale human evaluation or simplified set-ups such as multiple-choice QA. By contrast, our work introduces a comprehensive benchmark with automated metrics, involves user studies with experts across three scientific disciplines and develops new methodologies to train specialized open models. OpenScholar greatly outperforms previously introduced systems and shows superiority over human experts in five domains.<\/p>\n Benchmarks for scientific literature understanding<\/p>\n Several works have developed benchmarks to evaluate the abilities of models to understand scientific literature. Previous datasets, such as SciFact42<\/a>, QASPER76<\/a> and QASA43<\/a>, largely focus on single-paper settings, in which the necessary information to answer queries is contained within a single preselected paper. However, in real-world scenarios, experts often need to synthesize information from several papers to answer questions. To address this gap, ScholarQABench introduces newly annotated tasks that require reasoning across several papers. There are also scientific summarization tasks, such as Multi-XScience77<\/a>, in which models are provided with several papers and asked to generate summaries, typically based on the related work sections of those papers. However, in this work, we focus on scenarios in which the relevant papers are not specified in advance, making the task more challenging. Recently, Xu et al.13<\/a> introduced KIWI, a dataset containing 200 questions and human-verified or edited answers generated by state-of-the-art LLMs, with a focus on the natural language processing domain. KIWI also provides a set of relevant papers that models must consider. Although both KIWI and ScholarQABench feature multi-paper, information-seeking tasks, ScholarQABench includes both human-written answers and automatic evaluation pipelines. By contrast, KIWI focuses more on human evaluations and its reference answers are primarily model-generated.<\/p>\n","protected":false},"excerpt":{"rendered":"OpenScholar OpenScholar (detailed in Extended Data Fig.\u20091) is a new retrieval-augmented LM designed to ensure reliable, high-quality responses…\n","protected":false},"author":2,"featured_media":449480,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32],"tags":[19944,1159,11382,1160,79],"class_list":["post-449479","post","type-post","status-publish","format-standard","has-post-thumbnail","category-science","tag-computer-science","tag-humanities-and-social-sciences","tag-information-technology","tag-multidisciplinary","tag-science"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/449479","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/comments?post=449479"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/449479\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media\/449480"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media?parent=449479"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/categories?post=449479"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/tags?post=449479"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}