All four models also routinely relied on foreign, state-owned media as reliable sources of information. <\/p>\n
In 35% of responses to foreign policy questions, the chatbots cited state-controlled sources such as China\u2019s Global Times or CGTN, or Russia\u2019s RT. ChatGPT and Grok were the worst offenders, citing state-owned media 51% and 44% of the time, respectively.<\/p>\n
In many of the cases, the chatbots returned biased or inaccurate information with a confidence that was even more misleading, the study found. <\/p>\n
\u201cThe most professional-looking answers, backed by strongest-looking citations, were also the most likely to contain buried factual errors,\u201d Forum AI said in a statement, calling it one of the study\u2019s \u201csharpest findings\u201d.<\/p>\n
Chatbots often struggle with news accuracy, especially on breaking stories where there is limited information available online. AI models that power chatbots are often trained on wide swathes of data found on the open web, a notoriously untrustworthy source of facts and nuance.<\/p>\n
Campbell Brown, chief executive of Forum AI and a former head of news partnerships at Meta Platforms, said she is particularly concerned about the study\u2019s results given the looming US Midterm election cycle. <\/p>\n
Few people use chatbots for news today, but that number will undoubtedly increase over time as they continue to siphon queries that used to go to Google\u2019s search engine.<\/p>\n
Brown conducted the study in the hope of holding the model makers more accountable. The struggle with news accuracy may encourage them to prioritise these types of queries in the same way they put maths or coding-focused interactions first, she said.<\/p>\n
\u201cWe\u2019d welcome the opportunity to review the underlying data behind this report,\u201d an Anthropic spokesperson said. <\/p>\n
\u201cClaude is trained to be politically even-handed in its responses, and to treat opposing viewpoints with equal depth, engagement, and quality of analysis, without bias towards any particular ideological position.\u201d<\/p>\n
None of the other three model makers commented for this story.<\/p>\n
\u201cIndependent evaluation is important,\u201d Brown, who co-founded Forum AI last year, said. <\/p>\n
The start-up used its own AI model to grade the chatbot makers, building it with input from a range of industry experts who have spent decades studying foreign affairs and geopolitics.<\/p>\n
\u201cThe model companies are essentially grading their own homework,\u201d Brown continued. \u201cAnd it\u2019s really important that there be companies outside of the model, companies that are doing this work and sharing the results.\u201d<\/p>\n
Major social media platforms such as Meta and Google\u2019s YouTube have historically shied away from fact-checking, particularly for topics that are widely polarising and politically charged, claiming they don\u2019t want to be the arbiters of truth for the rest of the internet.<\/p>\n
Brown believes AI companies will be different.<\/p>\n
\u201cAt Meta, you\u2019re optimising for engagement. And if you\u2019re optimising for engagement, it\u2019s also hard to optimise for accuracy,\u201d she said. <\/p>\n
AI companies that sell their models to enterprise clients are in a different situation, Brown added. Those paying customers will expect accuracy as a baseline.<\/p>\n
\u201cI just think it\u2019s an entirely different product at the end of the day,\u201d she said.<\/p>\n
Sign up to Herald Premium Editor\u2019s Picks<\/a>, delivered straight to your inbox every Friday. Editor-in-Chief Murray Kirkness picks the week\u2019s best features, interviews and investigations. Sign up for Herald Premium here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"All four models also routinely relied on foreign, state-owned media as reliable sources of information. In 35% of…\n","protected":false},"author":2,"featured_media":438857,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[1443,3225,213649,224311,3017,224308,95,32955,167362,9556,138,4035,4036,7705,7797,224310,13573,3011,71,6301,65003,367,7710,9557,661,149872,111,43,139,69,1118,110864,17146,69009,9939,7235,666,383,9938,3040,102,224309],"class_list":["post-438856","post","type-post","status-publish","format-standard","has-post-thumbnail","category-business","tag-about","tag-according","tag-accuracy","tag-accurately","tag-ahead","tag-alphabets","tag-and","tag-answer","tag-anthropics","tag-artificial","tag-business","tag-chatbots","tag-chatgpt","tag-claude","tag-elections","tag-fairly","tag-forum","tag-four","tag-from","tag-gemini","tag-geopolitics","tag-google","tag-grok","tag-intelligence","tag-major","tag-midterms","tag-new-zealand","tag-news","tag-newzealand","tag-nz","tag-of","tag-openais","tag-questions","tag-sourcing","tag-states","tag-struggle","tag-struggling","tag-study","tag-united","tag-us","tag-with","tag-xais"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/438856","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=438856"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/438856\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/438857"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=438856"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=438856"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=438856"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}