![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-20-at-08.13.22-678x381.png\" "\\"Screenshot")
<\/p>\n<\/p>\n
The latest Vals Legal AI Report (VLAIR) has been published and focuses on legal research. It\u2019s certainly got attention because key players didn\u2019t take part and AI clearly \u2018beat\u2019 human lawyers. AL takes a look at what it all means.<\/p>\n
First, what is Vals? It\u2019s a US business working on the evaluation of AI systems. The focus here on legal research is just one of many projects they run. They brought in a group of legal AI companies, some human lawyers, and also tested against a general model. In this case the evaluated products were Alexi, Counsel Stack, Midpage, plus ChatGPT. At least one major vendor did take part, but then didn\u2019t want its results made public (more on that below).<\/p>\n
How does Vals compare human lawyer work with that of AI? They explained that they \u2018establish a baseline measure of the quality of work produced by the average lawyer, unaided by generative AI. To achieve this, we partnered with a US law firm who supplied lawyers experienced in conducting legal research for client matters. The lawyers were asked to answer the Dataset Questions based on the exact same instructions and context provided to the AI products.\u2019<\/p>\n
And in terms of how it was all carried out, the \u2018response collection took place over the first three weeks of July 2025. All questions were submitted to each product via API as a zero-shot prompt.\u2019 So, companies \u2013 and general models \u2013 may have improved since then, as three months is a long time in genAI. Also, the test was not about how people used AI tools and the application layers that come with them, it was really laser-focused on a straight question\/answer approach, without follow-ups.<\/p>\n
The overall finding is shown below, i.e. all the AI systems, including the general model, beat the human lawyers on 200 legal research question \u2018distributed across a range of question types typically encountered in private practice\u2019. They used three main criteria:<\/p>\n
\u2018- Accuracy: whether the response is substantively correct with no incorrect elements (50% of average weighted score)<\/p>\n
\u2013 Authoritativeness: whether the response is supported by citations to relevant and valid primary and\/or secondary law sources (40% of average weighted score)<\/p>\n
\u2013 Appropriateness: whether the response is easy to understand and could be immediately shareable with colleagues or clients (10% of average weighted score)\u2019.<\/p>\n
The AIs won on all three.<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-20-at-06.56.31-1024x579.png\")
<\/p>\n
AL View<\/p>\n
\u2018AI Is Better Than Lawyers At Legal Research\u2019<\/p>\n
OK, this is the big one that has gained the most attention. But, what does it mean? The \u2018[test group of lawyers] were allowed to use all research resources and tools at their disposal provided they were not generative AI-based\u2019. I.e. the human lawyers were also using digital databases packed full of legal information. \u00a0<\/p>\n
So, is this a surprise? On one level, no. Legal data stores are vast, complex, and can absorb huge amounts of time if lawyers are able to keep digging and digging. Given genAI\u2019s strong language understanding capabilities, and the ability to tap this vast sea of data very quickly, it isn\u2019t surprising that human lawyers were beaten here.<\/p>\n
Also, when it comes to things like summarization, it would again \u2013 because of genAI\u2019s language understanding skills \u2013 have been surprising if a test group of human lawyers had clearly beaten the AI.<\/p>\n
If the lawyers had been strongly incentivised to spend considerable time on each and every question, and perhaps had a supervising partner there to keep saying \u2018not good enough, go back and keep looking\u2019, or \u2018no, rephrase that, add more context, do it again\u2019, and provide other high-quality feedback, then one would have expected the test group of lawyers to do a lot better. But, they were not in that situation, it appears.<\/p>\n
OK, so, genAI tools are better at legal research tasks. What does this tell us? Well, it tells us that AI tools are good at handling tons of language. The questions, see below, didn\u2019t ask the AI tools to then take that information and go to court with it, stand in front of a judge, or build a litigation strategy with the client. So, we are still far from \u2018the end of lawyers\u2019. Although, it may make some law librarians a bit nervous.<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-20-at-07.21.09-1024x519.png\")
<\/p>\n
Moreover, is anyone really that sad that AI tools can provide very good legal research results? If AI tools can\u2019t do well on this type of database-focused task, then they wouldn\u2019t be much use for anything else.<\/p>\n
Overall, A) this is not a big surprise, and B) this is welcome news. And C) we can add the caveats that the lawyers were not heavily incentivised to keep working on these questions, nor does a distinct research output equate to sophisticated legal skills in court.<\/p>\n
That said, if you are a client reading this, and your law firm is charging you a lot for legal research, then you may want to have a conversation with them.<\/p>\n
\u2013<\/p>\n
\u2018The Big Players Didn\u2019t Play\u2019<\/p>\n
As noted, AL knows that at least one \u2018big player\u2019 took part and then decided it didn\u2019t want its results shown \u2013 which is their prerogative. However, this opens up a range of issues.<\/p>\n
Are studies valid if only a small number of vendors take part, especially if the big brands are not there? The short answer is that having the big legal data players in the study would have made it much more valuable to everyone, because those are the ones that most lawyers use. Simple as that.<\/p>\n
Why didn\u2019t the big players play ball? As with previous studies (by a range of groups), some companies have grumbled that the gap between the test and the publication is so great that the scores are not meaningful any longer, as three months is a long time in legal AI. That is a fair argument \u2013 although, the other vendors are all in the same boat. They could also argue that the tasks were better fitted to some companies and not their one. But, if you are in \u2018legal research\u2019 and the tasks are all about \u2018legal research\u2019, then surely this is a test you can do well at? One final reason is simply that the bigger brands have more to lose by comparing against other companies, even if they only fall short by 1% on a metric; and their top teams do not want that kind of message to go public\u2026.even if the end scores are only part of the value the big platforms provide. I.e. the application layer and all that goes with a sophisticated platform counts for a lot, and one metric in one study should not be seen as a final proof.<\/p>\n
Should the big players play ball? Well\u2026.yes. At the end of the day evals are only really useful if they test the tools that most lawyers mostly use \u2013 as well as the new kids on the block. Much as the ones tested are doing great work, we all know which tools law firms tend to use for legal research needs.<\/p>\n
Do Tests Like This Give Us A Full Picture?<\/p>\n
The reality is that a lawyer\u2019s work is not just \u2018find an answer to this specific question\u2019. The work of a lawyer is within an ever-complexifying world, with multiple layers, client interactions, battling the other side\u2019s lawyers, and much more. The ability to use AI to answer a legal research query is not the end of lawyers, not even close \u2013 not unless a client\u2019s main reason for hiring a law firm is to do legal research for them.<\/p>\n
Also, ChatGPT doing about as well as the legal specific tools can give a confusing impression. Legal AI tools will come with a dense application layer, providing a range of additional benefits, from UI\/UX made for legal needs, to RAG systems, to workflow management, to connecting to a dozen other legal \u2018skills\u2019 and having all of that available in one place, and connected directly to your DMS and other data stores.<\/p>\n
Could a law firm use just ChatGPT on its own for these types of question? Yes. As shown, the results are good. But, we should not ignore the points above. That said, it puts pressure on all legal AI providers to offer more than a \u2018point and shoot\u2019 capability. To add value above and beyond what a general model can do \u2013 especially now, as they widen their offerings \u2013 means to provide something special. And that\u2019s a good motivator for the sector as a whole. <\/p>\n
Overall, as noted by AL before, we need more than one study every now and then. Accuracy and its related values is a destination that we are travelling toward, and using a compass to get there. No test on its own is enough. We need a basket of evals. We also need to see evals as not so much as a test of individual companies, but as a statement of intent to the sector that lawyers want provable accuracy levels, and that those levels will keep going upwards.<\/p>\n
Final comments.<\/p>\n
AL asked Rayan Krishnan, of Vals, and also Tara Waters \u2013 the legal tech consultant, who is working with Vals, about A) the performance of general models vs legal AI tools, and B) about the limits of specific tasks as a comparator for what lawyers do more broadly. Here\u2019s what they told AL.<\/p>\n
Rayan \u2013 \u2018My initial take is that the underlying models have improved very quickly. In particular they have been able to 1) leverage data available on the web very well and 2) limit their own incidence of hallucinations. This seems to have led to generalist models that\u00a0can do legal research well. There\u00a0are still\u00a0some gaps, especially around their \u201cAuthoritative\u201d score or the sources they use, but this may continue to narrow.\u2019<\/p>\n
Tara \u2013 \u2018On the question of ChaptGPT\u2019s performance, I agree as the latest models have much more up-to-date training. And we\u2019ve seen the AI labs are starting to push their legal capabilities so one would think there is some proactive work happening there.\u00a0<\/p>\n
\u2018I personally didn\u2019t find it so surprising that ChatGPT was up there on accuracy; I was mildly surprised its sourcing hasn\u2019t yet been optimised for identifying proper legal sources first. TBH, that seems potentially an easy fix for the AI labs though (or I may be underestimating the complexity).<\/p>\n
\u2018However, and somewhat relevant to your second question, it\u2019s also important to note we didn\u2019t test \u201cresearch and prepare information for court filings\u201d types of questions \u2014 which is really where we\u2019ve seen the big issues in using public chatbots in the legal research context.<\/p>\n
\u2018On your second question, I think there is general industry agreement that accuracy is not the only factor relevant in assessing AI capabilities, overall quality or value. However, it\u2019s a huge challenge to assess the qualitative and user-led metrics at scale. I\u2019ve had a few rich discussions on LinkedIn about this, and it\u2019s something we hope to work towards solving.<\/p>\n
\u2018At the same time, having a clear benchmark of metrics like accuracy and authoritativeness allows us to shift the question from \u201cwhat does legal AI give us on top of the models\u201d to \u201cwhat do we want\/need legal or any AI to provide us with and who\u2019s got that on offer\u201d and probably also \u201c\u2026and what is that worth to us\u201d.\u2019<\/p>\n
\u2013<\/p>\n
Overall, a welcome study that gets us all engaged with the key issue of AI accuracy and encourages the legal AI companies to keep pushing forward. Thanks to Rayan and Tara, and the lawyers and companies that took part.<\/p>\n
\u2014<\/p>\n
The full report can be found here<\/a> (via main website).<\/p>\n \u2014<\/p>\n Legal Innovators Conferences in London and New York \u2013 November \u201925<\/p>\n If you\u2019d like to stay ahead of the legal AI curve then come along to\u00a0Legal Innovators New York<\/a>, Nov 19 + 20 and\u00a0also,\u00a0Legal Innovators UK<\/a>\u00a0\u2013 Nov 4 + 5 + 6, where the brightest minds will be sharing their insights on where we are now and where we are heading.\u00a0<\/p>\n Legal Innovators UK arrives first, with:\u00a0Law Firm Day on Nov 4th, then Inhouse Day, on the 5th, and then our new Litigation Day on the 6th.\u00a0<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/Screenshot-2025-06-18-at-09.17.37-1024x557.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/Screenshot-2025-06-18-at-09.16.05-1024x465.png\")
<\/a><\/p>\n