{"id":631297,"date":"2026-10-15T10:50:24","date_gmt":"2026-10-15T10:50:24","guid":{"rendered":"https:\/\/www.newsbeep.com\/nz\/631297\/"},"modified":"2026-10-15T10:50:24","modified_gmt":"2026-10-15T10:50:24","slug":"five-ai-chatbots-were-tested-on-accuracy-the-results-werent-good","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/nz\/631297\/","title":{"rendered":"Five AI chatbots were tested on accuracy. The results weren\u2019t good"},"content":{"rendered":"

\"Independent<\/p>\n

Want to bookmark your favourite articles and stories to read or reference later? Start your Independent Membership today.<\/p>\n

Join today<\/a><\/p>\n

\n Already a member?
\n Log in<\/a>\n <\/p>\n

Popular AI chatbots<\/a> are prone to making false claims, cite out of date figures, and answer the same question differently on different days, a new report into their accuracy has found.<\/p>\n

Chatbots<\/a> \u201cconfidently\u201d provided wrong responses to prompts, including one asserting that Sir Keir Starmer was still UK prime minister as late as September \u2013 weeks after he resigned<\/a>, the report from fact checking<\/a> charity Full Fact found.<\/p>\n

Its researchers asked the same questions to five widely used AI models every day over 10 weeks from mid-July in a bid to analyse the accuracy, consistency, and timeliness of the responses.<\/p>\n

\"ChatbotsChatbots including ChatGPT were asked 22 questions over ten months (Getty)<\/p>\n

Claude<\/a>, ChatGPT<\/a>, Gemini, DeepSeek<\/a>, and Grok, were asked 22 questions including \u201cDo I need ID to vote?\u201d and \u201cHow many migrants arrived by small boats in the first half of 2026?\u201d.<\/p>\n

The report said the chatbots often gave correct answers and \u201cresisted well-established misinformation\u201d. But for some questions, the models calculated incorrect figures themselves instead of citing official statistics, or presented outdated figures as current.<\/p>\n

Full Fact cites an example where chatbots were asked how many migrants arrived in the UK by small boat in the first half of 2026. Most gave correct figures, but on 4 August, Gemini wrongly reported 12,210 arrivals. Fact checkers said it reached the figure by subtracting the 2,316 arrivals recorded in July from a total of 14,526 reported as of 1 August. Although the maths was correct, the model forgot to exclude 1 August meaning the resulting figure covered January to June, as well as 1 August.<\/p>\n

On another occasion, chatbots were asked when England last won a major football trophy. Several models identified the correct answer, 2025 \u2013 when the Lionesses won the Euros \u2013 on some days but incorrectly answered 2022 on others.<\/p>\n

DeepSeek also reported that Sir Keir Starmer was still prime minister on 15 September, despite Andy Burnham<\/a> taking over in July. <\/p>\n

\"ScreenshotsScreenshots from the report show how chatbots gave different responses to the same question on different days (Full Fact)<\/p>\n

When asked how much money asylum seekers are given to live on, several chatbots cited a weekly rate of \u00a38.86, which was superseded by \u00a39.95 in June 2025. Gemini and Claude both gave the outdated figure outright in some responses.<\/p>\n

The examples are one of a number which demonstrate how chatbots can confidently give incorrect statistics, and fail to attribute relevant sources, the charity said.<\/p>\n

Chris Morris, chief executive of Full Fact, said that while models did not necessarily \u201cpeddle misinformation,\u201d responses were often \u201cincomplete, lacking context, or poorly sourced\u201d.<\/p>\n

“Millions of people increasingly rely on AI chatbots for timely responses to factual questions \u2014 what do I need, what am I entitled to, has a rule changed?\u201d he said. \u201cOur findings show the answer comes back in the same confident tone whether it is correct, out of date, or describes something that has not yet happened.<\/p>\n

\"DeepSeekDeepSeek said Sir Keir Starmer was the prime minister nearly two months after Andy Burnham had been sworn in (Full Fact)<\/p>\n

“That is the real problem here. These models are not necessarily peddling misinformation, but their responses can be incomplete, lack context or be poorly sourced.\u201d<\/p>\n

He called on leading AI companies to ensure models check live, authoritative sources for time-sensitive facts and be able to account for the sources they cite.<\/p>\n

“The bosses of these AI companies say they are happy for others to mark their homework, so we have started doing it for them,\u201d he continued. <\/p>\n

\u201cOur aim is to provide a shared evidence base on how leading AI models behave in practice as they evolve. We want this extraordinary technology to work for everyone, in the public interest, and independent assessment is one way to help make that happen.\u201d<\/p>\n

OpenAI said improving factual accuracy is a major focus, and that it continues to reduce errors with newer models.<\/p>\n

Anthropic<\/a>, Google, DeepSeek, and Grok have been contacted for comment.<\/p>\n","protected":false},"excerpt":{"rendered":"Want to bookmark your favourite articles and stories to read or reference later? Start your Independent Membership today.…\n","protected":false},"author":2,"featured_media":631298,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[365,363,364,111,139,69,145],"class_list":["post-631297","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-new-zealand","tag-newzealand","tag-nz","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/631297","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=631297"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/631297\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/631298"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=631297"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=631297"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=631297"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}