{"id":32583,"date":"2025-07-29T19:58:15","date_gmt":"2025-07-29T19:58:15","guid":{"rendered":"https:\/\/www.newsbeep.com\/ca\/32583\/"},"modified":"2025-07-29T19:58:15","modified_gmt":"2025-07-29T19:58:15","slug":"big-businesses-are-creating-their-own-tests-to-find-the-best-ai-models","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/ca\/32583\/","title":{"rendered":"Big businesses are creating their own tests to find the best AI models"},"content":{"rendered":"<p>TORONTO \u2014 Major firms including Manulife and RBC are now using their own internal tests to better evaluate the performance of new AI models in an attempt to cut through the industry hype.<\/p>\n<p>While AI developers regularly tout the performance of their models on popular third-party evaluations, executives say such assessments don\u2019t really show whether the systems are well-suited for their business needs. \u201cThe benchmarks can actually lead you astray,\u201d according to Jodie Wallis, global chief AI officer at Manulife.\u00a0<\/p>\n<p>Manulife\u2019s tech team has built its own set of tests based on 47 actual use cases in areas like customer service and risk assessment, Wallis said. The program lets the firm plug the latest releases from AI labs into its existing tools, using its own data. That cuts the time to evaluate a new model down from weeks to minutes, and allows it to more quickly adapt when better technology becomes available, she claimed. <\/p>\n<p>Talking Points<\/p>\n<p>Large firms like Manulife and RBC have developed their own benchmarks for testing AI models to work out how well-suited they are for their businesses<br \/>\nTech firms often tout how well their models perform against popular benchmarks, but mathematical or knowledge exams don\u2019t always reflect how well they do on real business tasks<\/p>\n<p>Model makers <a href=\"https:\/\/www.anthropic.com\/news\/claude-3-5-sonnet\" rel=\"nofollow noopener\" target=\"_blank\">typically<\/a> <a href=\"https:\/\/cohere.com\/blog\/command-a\" rel=\"nofollow noopener\" target=\"_blank\">cite<\/a> the performance of their products on popular tests like MMLU, which <a href=\"https:\/\/arxiv.org\/abs\/2009.03300\" rel=\"nofollow noopener\" target=\"_blank\">tests<\/a> accuracy across different tasks; MATH, which <a href=\"https:\/\/arxiv.org\/abs\/2103.03874\" rel=\"nofollow noopener\" target=\"_blank\">assesses<\/a> mathematical problem-solving; or BIRD-SQL, which <a href=\"https:\/\/bird-bench.github.io\/\" rel=\"nofollow noopener\" target=\"_blank\">checks<\/a> coding ability. Developers also <a href=\"https:\/\/www.semafor.com\/article\/03\/15\/2023\/how-gpt-4-performed-in-academic-exams\" rel=\"nofollow noopener\" target=\"_blank\">watch<\/a> AI models\u2019 scores on university admissions exams.<\/p>\n<p>Manulife doesn\u2019t need its tools to ace the LSAT or the MCAT, but it does, Wallis said, want models that deliver the most accurate results at the lowest cost for its in-house applications.<\/p>\n<p>New models out-performing their predecessors by a few percentage points isn\u2019t that important for large firms adopting AI, said Foteini Agrafioti, senior vice-president of data and AI at RBC. Rather than raw power, the bank needs models that it can use securely and quickly scale across the company.<\/p>\n<p>RBC plans to <a href=\"https:\/\/thelogic.co\/briefing\/cohere-launches-ai-agent-builder-system-for-businesses-starting-with-rbc\/\" rel=\"nofollow noopener\" target=\"_blank\">use<\/a> Cohere\u2019s technology for most of its generative tools. The bank wants to standardize and centralize its AI platform, the same way it has with software for other functions like cybersecurity or engineering, Agrafioti said. A single system makes it easier to keep data safe and transfer tools between different parts of the business, she added.\u00a0\u00a0<\/p>\n<p>RBC tested technology from other firms, including OpenAI, Agrafioti said, but the bank wasn\u2019t willing to send client data to a system hosted on another company\u2019s cloud. So, RBC bought access to North, a Cohere product that lets users launch AI agents powered by the Toronto firm\u2019s large language models. It will run Cohere\u2019s technology on its own servers; the bank has built what it <a href=\"https:\/\/rbcborealis.com\/applications\/lumina-platform\/\" rel=\"nofollow noopener\" target=\"_blank\">claims<\/a> is Canada\u2019s largest cluster of the graphics processing units used to power AI.\u00a0<\/p>\n<p>\t\t\t\t\tRelated Articles<\/p>\n<p>\t<a href=\"https:\/\/thelogic.co\/news\/chatbot-arena-meta-google-ai-models\/\" rel=\"nofollow noopener\" target=\"_blank\"><br \/>\n\t<img loading=\"lazy\" width=\"768\" height=\"512\" src=\"https:\/\/www.newsbeep.com\/ca\/wp-content\/uploads\/2025\/07\/Meta_-Founder_CEO_Mark_Zuckerberg-LlamaCon_2025-AI_developer_conference-Menlo_Park_Calif-April_2025-.jpeg\" class=\"attachment-medium_large size-medium_large wp-post-image\" alt=\"Meta founder and CEO Mark Zuckerberg sitting on a white chair, holding a microphone, smiling, against a blue background.\" decoding=\"async\"  \/>\t<\/a><\/p>\n<p>\t<a href=\"https:\/\/thelogic.co\/news\/best-ai-model-chatgpt-llama-vector-institute\/\" rel=\"nofollow noopener\" target=\"_blank\"><br \/>\n\t<img loading=\"lazy\" width=\"768\" height=\"512\" src=\"https:\/\/www.newsbeep.com\/ca\/wp-content\/uploads\/2025\/07\/Sam_Altman_OpenAI-San_Francisco-Nov_2023-P_Eric_Risberg-CP169401574-1920x1280-1-768x512.jpeg\" class=\"attachment-medium_large size-medium_large wp-post-image\" alt=\"OpenAI CEO Sam Altman speaks at the Asia-Pacific Economic Cooperation CEO Summit in San Francisco, in November 2023.\" decoding=\"async\"  \/>\t<\/a><\/p>\n<p>RBC and Cohere are co-developing a version of North that meets the bank\u2019s security and regulatory requirements. RBC employees building new AI applications that touch sensitive data must use North, which is rolling out to developers at the company this summer. In March, Canada\u2019s biggest bank <a href=\"https:\/\/www.rbc.com\/investor-relations\/_assets-custom\/pdf\/investorday2025_presentation.pdf\" rel=\"nofollow noopener\" target=\"_blank\">announced<\/a> it\u2019s aiming to generate up to $1 billion in earnings using AI by 2027. \u201cIt\u2019s graduating out of experiments,\u201d Agrafioti said, to become \u201cmainstream.\u201d<\/p>\n<p>Betting on Cohere as RBC\u2019s main generative AI provider comes with risks, Agrafioti said, since other AI firms might advance the technology faster. The bank will monitor those alternatives and give staff access to the latest models, she said. Still, she\u2019s keen to avoid the distraction of flashy new AI models. \u201cWe have something in our hands that does the job we need it to do really well, and we\u2019re going to use that,\u201d she said.<\/p>\n<p>While big businesses may have the data, the technical staff and use cases to test AI models, smaller firms rely on third-party benchmarks such as LLM Arena and MMLU, including some that researchers have <a href=\"https:\/\/thelogic.co\/news\/chatbot-arena-meta-google-ai-models\/\" rel=\"nofollow noopener\" target=\"_blank\">accused<\/a> tech firms of trying to game.\u00a0<\/p>\n<p>Toronto\u2019s Vector Institute is trying to provide a more independent analysis. In April, it <a href=\"https:\/\/thelogic.co\/news\/best-ai-model-chatgpt-llama-vector-institute\/\" rel=\"nofollow noopener\" target=\"_blank\">released<\/a> its first \u201cstate of evaluation\u201d study, which ran 11 models through 16 tests. \u201cLarge companies have built this infrastructure to do evaluations internally,\u201d said Deval Pandya, the institute\u2019s vice-president of AI engineering. \u201cBut most of it is not openly accessible.\u201d <\/p>\n","protected":false},"excerpt":{"rendered":"TORONTO \u2014 Major firms including Manulife and RBC are now using their own internal tests to better evaluate&hellip;\n","protected":false},"author":2,"featured_media":32584,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[62,276,277,49,48,24908,17042,64,61,24909],"class_list":["post-32583","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-ca","tag-canada","tag-manulife","tag-rbc","tag-tech","tag-technology","tag-vector-institute"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/32583","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/comments?post=32583"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/32583\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media\/32584"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media?parent=32583"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/categories?post=32583"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/tags?post=32583"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}