{"id":642993,"date":"2026-10-27T17:30:19","date_gmt":"2026-10-27T17:30:19","guid":{"rendered":"https:\/\/www.newsbeep.com\/nz\/642993\/"},"modified":"2026-10-27T17:30:19","modified_gmt":"2026-10-27T17:30:19","slug":"confident-science-papers-sway-ai-judges-regardless-of-merit-unite-ai","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/nz\/642993\/","title":{"rendered":"Confident Science Papers Sway AI Judges \u2013 Regardless of Merit \u2013 Unite.AI"},"content":{"rendered":"

Confidence is a distinct advantage in evolution<\/a> \u2013 and, according to a new research paper from Germany, also an advantage when a scientific paper is getting reviewed by Large Language Models (LLMs<\/a>) instead of an author\u2019s peers (an increasingly frequent necessity, as the volume of science papers balloons<\/a> thanks to the productivity gains<\/a> under AI).<\/p>\n

The new study finds that AI (i.e., non-human) reviewers consistently give higher scores to scientific papers that are written in confident, assertive language \u2013 even when the underlying research is identical to papers with a less confident tone. In some cases, changing nothing but the wording proves enough to turn a rejection into an acceptance.<\/p>\n

The authors state:<\/p>\n

\u2018Our experiments reveal substantial preferences of LLM reviewers toward [papers] using high-confidence language. Crucially, these preferences frequently result in decision flips across the accept<>reject boundary, directly affecting review outcomes. <\/p>\n

\u2018Beyond scores, we find that aspects raised in reviews vary in frequency and sentiment depending on linguistic features of the paper, potentially influencing final decisions.\u2019<\/p>\n

It\u2019s ironic, because the syndrome that the authors identify is very similar to the way<\/a> AI can so often be \u2018confidently wrong\u2019<\/a> when responding to our own prompts and queries; except here, the decision is human, and the individual and collective responsibility real, whether or not the contributing scientists behind these papers really believe their own claims.<\/p>\n

If AI increasingly decides which scientific papers get published, scientists may learn to \u2018sound more confident\u2019 as a competitive tactic, perhaps most particularly when their submissions are flawed; and this could allow weaker science an undeserved foothold in the ever-diminishing pool of available attention.<\/p>\n

Bluster Towards Victory<\/p>\n

According to the study<\/a>*, AI reviewers were more likely to praise the methods, theories and scientific contributions of confidently-written papers, even when these were identical to those in less assertive versions.\u00a0In the authors\u2019 tests, 504 scientific papers were rewritten in different styles, and assessed by six AI models, including GPT-5.5<\/a>, Claude Sonnet 4.6<\/a> and Gemini 3.5 Flash<\/a>. It was necessary to generate artificial\/amended versions of papers, so that the eccentricities and quirks of real submissions did not skew the results:<\/p>\n

\"Examples<\/p>\n

Examples of the same scientific passage rewritten in four different styles. Colored highlights identify changes intended to make the writing sound more or less confident, or more like academic English written by native or non-native speakers. However, the underlying findings and figures remain unchanged. Source<\/a><\/p>\n

The papers were drawn from submissions to the 2026 ICLR conference, comprising equal numbers of accepted and rejected papers. Claude Opus 5<\/a> then rewrote each submission to sound more or less confident\u2020, keeping the scientific content unchanged, and thus allowing differences in the AI reviews to be attributed to writing style, i.e., a varied level of confidence expressed.<\/p>\n

In the results, AI reviewers gave higher scores to scientific papers written in confident language, even when the evidence was identical. Claude Sonnet 4.6 showed the strongest bias, favoring confident versions seven times more often than cautious ones. In 21% of cases, it recommended accepting a confidently-written paper, while rejecting a more cautiously-expressed version of the same paper.<\/p>\n

The new study states:<\/p>\n

\u2018Our findings raise important concerns for the use of LLMs in peer review. If content-equivalent papers receive different scores and review arguments solely because of linguistic properties, reviewer judgments do not exclusively reflect scientific quality. <\/p>\n

\u2018LLM reviewers should therefore be stress-tested on content-preserving rewrites to determine whether linguistic changes systematically affect their assessments.\u2019<\/p>\n

The authors go on to suggest that these tests should explicitly examine both the papers\u2019 scores and their written reviews, to establish whether confident language is being mistaken for stronger scientific evidence.<\/p>\n

Method<\/p>\n

A total of 504 scientific papers were selected from submissions to the 2026 International Conference on Learning Representations (ICLR). The sample was divided equally between accepted and rejected papers, with 24 submissions drawn from each of the conference\u2019s 21 research categories:<\/p>\n

\"The<\/p>\n

The three-stage testing method. Scientific papers were rewritten to vary apparent English-language proficiency or confidence, while preserving their findings; the rewrites were checked for changes in style and meaning before being assessed by AI reviewers under identical conditions; and differences in review scores were then compared.<\/p>\n

Each paper was anonymized and rewritten using Claude Opus 5, producing alternative versions designed to test both the confidence with which claims were expressed, and the apparent English-language proficiency of the writer.\u00a0For the confidence tests, statements were made either more assertive or more cautious. Words suggesting certainty, such as \u2018clearly\u2019 and \u2018certainly\u2019, were used more frequently in \u2018confident\u2019 versions, while qualifying expressions such as \u2018may\u2019 and \u2018suggests\u2019 were favored in more circumspect drafts.<\/p>\n

In every case, the scientific claims, evidence, results, technical terminology and overall structure remained unchanged. The authors note that the rewriting instructions were based on established linguistic research, rather than on general requests to imitate particular writing styles.<\/p>\n

Checking the Rewrites<\/p>\n

For checks, each rewrite was compared with its original using semantic similarity<\/a>, which assesses whether passages retain their meaning; and TF-IDF<\/a> similarity (which measures the preservation of vocabulary and terminology). Both scores approached 1, confirming strong similarity to the originals:<\/p>\n

\"Checks<\/p>\n

Checks of the rewritten papers. Confident versions used more assertive language, while cautious versions expressed more uncertainty. Scores near 1 confirm strong similarity to the originals, and colored highlights mark the higher values for each targeted measure.<\/p>\n

Six AI reviewers were tested: proprietary models GPT-5.5; Claude Sonnet 4.6; and Gemini 3.5 Flash; and open-weight models Gemma 3<\/a>; Qwen3 VL<\/a>; and GLM 4.7<\/a>. These were chosen partly because they all have pre-2026 knowledge cutoff dates<\/a>, reducing the risk of prior knowledge of the papers\u2019 acceptance decisions.<\/p>\n

Each rewritten paper was reviewed independently using conference-style guidelines. To establish the extent to which scores might vary between repeated assessments, ten randomly-selected papers were reviewed ten times by each model.<\/p>\n

Results<\/p>\n

The tests conducted examined whether the six selected AI reviewers would consistently favor particular writing styles, and compared papers written in \u2018confident\u2019 or \u2018cautious\u2019 language, and papers resembling academic English written by native or non-native speakers\u2020.<\/p>\n

\"Results<\/p>\n

Results comparing AI review scores for 504 scientific papers rewritten in different styles. Preference percentages show which version scored higher, with ties in parentheses; ratios show how much more often one version was favored; and decision flips show how often changing the wording reversed an acceptance or rejection. Bold statistical values indicate significant differences. The relevant section is the lower one, as the upper hypothesis is not being covered in this article (see footnotes).<\/p>\n

Confident writing was consistently favored by all six AI reviewers. The strongest preference was shown by Claude Sonnet 4.6, which, as mentioned earlier, awarded higher scores to \u2018confident\u2019 drafts seven times more often than to \u2018cautious\u2019 ones.<\/p>\n

A paper was considered accepted if its score reached 4.5 out of 10, though changes in writing style sometimes pushed the same paper above or below this threshold, reversing the decision; and these reversals generally favored confident writing.<\/p>\n

Claude Sonnet 4.6 performed particularly poorly, accepting the \u2018confident\u2019 version while rejecting the \u2018cautious\u2019 version in 21% of cases. The reverse occurred in just 2.8% of cases.<\/p>\n

The effect was also evident in the written reviews, where AI reviewers were more likely to praise a paper\u2019s methods, theories and scientific contributions if its claims were expressed confidently. Since the research itself had not changed, these more favorable assessments could only be attributed to differences in presentation and tone.<\/p>\n

Conclusion<\/p>\n

Every time humans use AI to (albeit inadvertently) increase the signal-to-noise-ratio, AI is trialed as the solution \u2013 here, to cope with the great increase in Artificial Intelligence research paper submissions; a surge that has increased the gatekeeping power and parameters of Arxiv<\/a> lately. Yet it must be clear by now that frontier LLMs, and all the tiers beneath them, still have far too many eccentricities to be generically entrusted as judges, except in the most rigidly parametrized circumstances.<\/p>\n

In such cases, turning smaller-scale LLMs into dedicated judging tools via fine-tuning<\/a> or LoRAs<\/a> would probably be a better idea. However, that entails at least a little extra effort, if not a little extra investment, whereas \u2018vanilla\u2019 local installs or API calls are quicker, and relatively cheap.<\/p>\n

This rather tersely-written paper may nonetheless be of more than average interest to the LLM end-user, since they will already be familiar with \u2018enthusiasm\u2019 as a bugbear that they must maneuver round in order to work effectively with AI; and, if this and similar LLM issues cannot yet be fixed, they can at least be accounted for.<\/p>\n

Before we take too derisive a tone about this kind of facile equivalence in the estimation of AI, it\u2019s worth remembering that these systems are as lost as the rest of us when it comes to epistemology \u2013 the search for definitive authority; we too are, historically, very prone to mistake the signal for the thing itself.<\/p>\n

\u00a0<\/p>\n

* Titled \u2018Style over Substance? Linguistic Bias in LLM-based Peer Review\u2019<\/p>\n

\u2020 Or to resemble academic English written by native or non-native speakers \u2013 a parallel hypothesis that the paper was not able to prove as effectively, and which is not addressed in this article. <\/p>\n

First published Tuesday, October 27, 2026<\/p>\n","protected":false},"excerpt":{"rendered":"Confidence is a distinct advantage in evolution \u2013 and, according to a new research paper from Germany, also…\n","protected":false},"author":2,"featured_media":642994,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[111,139,69,147],"class_list":["post-642993","post","type-post","status-publish","format-standard","has-post-thumbnail","category-science","tag-new-zealand","tag-newzealand","tag-nz","tag-science"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/642993","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=642993"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/642993\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/642994"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=642993"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=642993"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=642993"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}