Pick a verdict from a 4-bucket rubric<\/p>\n
The five models (GPT<\/a>-5.4, Claude Opus 4.7<\/a>, Gemini 3 Pro<\/a>, Gemini 3 Pro<\/a> + Search, Sonar Pro<\/a>) were each given the same real-world claim and asked to pick a verdict from a 4-bucket rubric<\/a> (True \/ Mostly True \/ Misleading \/ False). Because only one bucket can be correct per claim, any disagreement among the panel means at least one model is label-inconsistent.<\/p>\n According to Lenz, the \u201csplit across these five models is intentional\u201d because it covers the spread of inference modes that are common in production AI systems.<\/p>\n How many types of inference are there?<\/p>\n Spanning from<\/a> latency-sensitive inference to throughput-aware, resource-constrained and scalable inference, inference is typically divided into low-latency high-throughput inference (e.g. for interactive chatbots) and offline or batch inference, where processes accumulate data before it is subsequently analyzed, once optimized for cost.<\/p>\n \u201cUnlike the standard benchmark questions, the models have not seen these claims during training \u2014 i.e., it\u2019s a fresh real-world corpus across science, healthcare, politics, law, and other domains.\u201d<\/p>\n Research informing the May 21 paper was led by Kosta Jordanov<\/a>, founder of Lenz and co-founder of Wiser<\/a>, an IT consulting and software engineering group headquartered in Sofia, Bulgaria.\u00a0\u00a0<\/p>\n Jordanov tells The New Stack that the claims his team used in the research are real claims that users have fact-checked on Lenz since February 15, 2026.\u00a0<\/p>\n A fresh, real-world corpus of data<\/p>\n \u201cWe\u2019ve excluded private claims, near-duplicate claims, and any claims containing personally identifiable information (PII),\u201d Joranov says. \u201cThe interesting thing about this corpus is that, unlike the standard benchmark questions, the models have not seen these claims during training \u2014 i.e., it\u2019s a fresh real-world corpus across science, healthcare, politics, law, and other domains on topics that people care about and fact-check.\u201d\u00a0<\/p>\n Beyond the 67% dissent metric, 34% of the claims are substantially disagreed on <\/p>\n Beyond the 67% dissent metric, 34% of the claims are substantially disagreed on (2+ buckets apart), and 21% are polar opposites (at least one model says False and at least one says True). At this level, we can start to see the path from dissent to disagreement having a real impact on live production AI systems and tools.<\/p>\n \u201cIf a software engineering team operates a system where legal, financial, or reputational risk is involved \u2013 and it delivers untrue or hallucinated content to users, you should think about the ways in which you validate the AI-generated content before it reaches users.\u201d\u00a0\u2014Kosta Jordanov.<\/p>\n What should AI developers think about this disconnect?<\/p>\n The practical takeaway is that on real-world claims, a single frontier LLM gives one opinion from a visibly unstable distribution. A second model often gives another.\u00a0<\/p>\n \u201cFor many applications, that\u2019s fine,\u201d Joranov clarifies. \u201cBut if a software engineering team operates a system where legal, financial, or reputational risk is involved \u2014 and it delivers untrue or hallucinated content to users \u2014 you should think about the ways in which you validate the AI-generated content before it reaches users.\u201d\u00a0<\/p>\n The question arises, then, why do frontier models converge confidently at True\/False poles but fracture badly on middle-ground verdicts? Unfortunately, that\u2019s a hard question to answer based on this research. One hypothesis Joranov puts forward is that the Mostly True and Misleading categories are a bit more ambiguous than the True and False categories.\u00a0<\/p>\n \u201cWhat we measured, though, is that some models use the middle buckets way less often than others \u2013 Gemini is quite \u2018confident\u2019 and classified only 6% of the claims in the two middle buckets vs. 45% for Opus 4.7,\u201d he says.<\/p>\n Is Anthropic especially out of line?<\/p>\n Looking at the potential howlers here, if Claude Opus 4.7<\/a> (which had received early criticism<\/a>) aligned with the peer majority least often at 70%, should that concern Anthropic?<\/p>\n \u201cNot necessarily,\u201d clarifies Joranov. \u201cOur limited preliminary research shows that the majority is often wrong, and sometimes we see wrong unanimous verdicts; i.e., having a different opinion than the majority does not necessarily mean being wrong.\u201d<\/p>\n This research does not use any \u201cground truths\u201d (indisputable real-world facts that have been widely validated and verified) and only measures the differences between the models\u2019 verdicts. It cannot answer which model is correct for which claim.<\/p>\n \u201cOur analysis [of LLM accuracy] reveals that apparent convergence in benchmark accuracy can conceal deep epistemic divergence.\u201d \u2013 Cornell University\u2019s Yang & Wang.<\/p>\n Other studies in this space<\/p>\n Academic and commercially underpinned model research appears to be turning to this space right now. A study<\/a>\u00a0by Eddie Yang and Dashun Wang at Cornell University\u00a0published in February notes that benchmarks underpin how progress in large language models (LLMs) is measured and trusted.\u00a0<\/p>\n \u201cYet our analyses reveal that apparent convergence in benchmark accuracy can conceal deep epistemic divergence<\/a>. Using two major reasoning benchmarks \u2014 MMLU-Pro and GPQA \u2014 we show that LLMs achieving comparable accuracy still disagree on 16-66% of items, and 16-38% among top-performing frontier models,\u201d wrote Yang & Wang in February.<\/p>\n Humans in the loop next<\/p>\n Joranov confirms that this analysis is the first step.\u00a0<\/p>\n \u201cWe do plan a follow-up where we measure the models against human-provided labels, and also measure the source-based multi-step multi-model Lenz pipeline against those labels and against the frontier models,\u201d Joranov says. \u201cThe time-consuming part is the methodologically correct labeling by human experts in all of those domains, but we aim to publish in the coming months.\u201d<\/p>\n This report concluded with a statement to explain that the point of this work isn\u2019t to create a leaderboard. <\/p>\n The point is to map the structure of disagreement, i.e., where do frontier panels systematically diverge from a human consensus, where does Lenz diverge from both, how each individual model and Lenz align with the same human reference, and what categories of claims drive each kind of divergence (rubric ambiguity, temporal framing, domain specialization, calibration drift).<\/p>\n
![](\"https:\/\/www.newsbeep.com\/ie\/wp-content\/uploads\/2026\/05\/684dae45-cropped-e991646b-06_rpa_inline_01_bridgwater-1-1-300x234-1.jpg\"\/)
<\/p>\n