Physics research is more than getting a number right. A researcher decides what to investigate, picks a method, checks that it works, and explains why the evidence matters. FrontierPhysics tests that whole chain. Physicists turned research they had conducted themselves into open-ended briefs, and agents are judged on both the resulting work and the route to it. Existing evaluations often isolate subtasks or hand the agent a published paper that already specifies the problem and much of its solution. [p2] Under this protocol, even the best tested setup passed fewer than three in ten trials. That is a benchmark result, not a claim that agents cannot do useful physics.<\/p>\n
![\"Horizontal](\"https:\/\/www.newsbeep.com\/ie\/wp-content\/uploads\/2026\/11\/01a1aa18-bf27-77b4-83e0-93d5f155f5bf.png\" "\\"Horizontal")
Overall Pass@1 (percent of trials passing) for each tested configuration, with approximate 95% confidence intervals. Claude Opus 5.5 is highest at 27.4%, followed by GPT-6 Astra at 23.2%; every displayed pass rate is below 30%, and some are zero. A pass requires all verifier tests and all blocking rubric criteria, not merely a high qualitative score.<\/p>\n
The tasks come from 42 physicists, who contributed work based on their own research. Of 182 submissions across nine physics domains, 56 tasks were accepted. [p5] A task gives the agent a high-level prompt, relevant materials, and the research problem, but not a complete method. [p3] The agent investigates the literature, forms a plan, carries out the research, validates its results, and submits structured outputs plus a manuscript. Task materials and workflows vary, so not every task involves the same kind of data, experiment, or simulation. [p2]<\/p>\n
Grading has two parts. Deterministic scripts check structured outputs against reference values, while a reviewer agent uses expert-authored rubrics to assess the manuscript and the trajectory of the work. [p5] Rubrics contain binary blocking criteria, which act as gates, and weighted non-blocking criteria describing other qualities. A trial passes only if every deterministic test and every blocker passes. “Publishable” here is therefore an operational standard defined by this protocol, not a judgment that a human journal would accept the manuscript.<\/p>\n
![\"Three-panel](\"https:\/\/www.newsbeep.com\/ie\/wp-content\/uploads\/2026\/11\/01a1aa18-c11a-79fc-93f8-c2225c9d74b1.png\" "\\"Three-panel")
Panel (a) shows curation: original research, quality review, subfield peer review, review of three agent runs for failure modes, then acceptance (56 of 182 submissions). Panel (b) shows an example task, rydberg-ising-critical-point, separating agent-visible materials from hidden grader and reference files. Panel (c) shows a sandboxed agent producing outputs and paper.pdf, followed by deterministic verification and rubric-based manuscript review; both gates determine pass or fail.<\/p>\n
The closest contrast is PRBench<\/a>, which asks agents to reproduce results from published physics papers, supplying the paper itself. FrontierPhysics asks researchers to turn their own projects into high-level tasks without specifying the solution path, and it also grades the manuscript and literature work. [p2] PRBench is a useful predecessor: its authors also evaluate end-to-end work, execution, and failure modes. NatureBench<\/a> also withholds source methods, but its tasks are built from published papers and it focuses on performance against published results rather than grading a manuscript and literature synthesis. The difference lies in task provenance and what gets evaluated.<\/p>\n Each configuration attempted all 56 tasks three times, giving 168 trials. Pass@1 is the share of those trials that pass, not the share of tasks ever solved. The top configuration, Claude Opus 5.5 in Claude Code, passed 46 of 168 trials, a Pass@1 of 27.4%. [p9] Its approximate 95% binomial interval is \u00b16.7 percentage points. A task-clustered bootstrap interval, which accounts for the three related trials per task, is wider: 17.9% to 36.9%. [p34] [p10]<\/p>\n To see why trials fail, task contributors and the admin team manually examined the trajectories and outputs of all 168 trials for one configuration, GPT-6 Astra xhigh. Among its 143 failed trials, lack of rigorous thinking was tagged in 48% and missing expert intuition in 34%. [p8] Tags can overlap, so percentages do not sum to 100%. Rigor failures include circular self-checking and missed edge cases. Intuition failures include not identifying the important error source while debugging. Literature problems were tagged in 27%, such as missing key sources or failing to support claims with them. Wrong physics was tagged in 16% and execution mistakes in 4%, so physics errors do occur; they are simply not the largest category in this cohort. [p10]<\/p>\n The review also exposed grading errors. Nine of the 168 outcomes were false failures. Correcting eight false-negative and one false-grading outcome changes the originally recorded pass tally for this configuration from 25\/168 (14.9%) to 34\/168 (20.2%). [p7] That is a manually corrected tally; the reported final rate in Table 2 is 19.6% after reruns and regrading, a different stage of the same bookkeeping. The authors caution that other models may fail in different ways, and that only GPT-6 Astra’s false negatives were accurately tagged, so this failure distribution should not be assumed for other models or harnesses. [p10]<\/p>\n A credible test of research needs to check not only whether outputs are correct but whether the route to them is adequately reasoned and scientifically communicated. [p10] FrontierPhysics makes that larger target measurable through researcher-authored, open-ended tasks and two complementary grading gates.<\/p>\n","protected":false},"excerpt":{"rendered":"Physics research is more than getting a number right. A researcher decides what to investigate, picks a method,…\n","protected":false},"author":2,"featured_media":676745,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[24],"tags":[61,60,248,82],"class_list":["post-676744","post","type-post","status-publish","format-standard","has-post-thumbnail","category-physics","tag-ie","tag-ireland","tag-physics","tag-science"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/posts\/676744","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/comments?post=676744"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/posts\/676744\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/media\/676745"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/media?parent=676744"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/categories?post=676744"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/tags?post=676744"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"Horizontal](\"https:\/\/www.newsbeep.com\/ie\/wp-content\/uploads\/2026\/11\/01a1aa18-bf98-7b87-bbd1-6adebb6e3053.png\" "\\"Horizontal")
Failure modes tagged in the manually reviewed GPT-6 Astra xhigh trials, as a percentage of the 143 failed runs (with counts): rigor 48% (68), intuition 34% (48), literature 27% (39), instruction-following 26% (37), wrong physics 16% (23), execution mistakes 4% (6). Bar colors distinguish whether a failure was flagged by the verifier only, by both verifier and blockers, or by blockers only. Tags overlap.<\/p>\n