{"id":37731,"date":"2025-09-23T01:54:09","date_gmt":"2025-09-23T01:54:09","guid":{"rendered":"https:\/\/www.newsbeep.com\/nz\/37731\/"},"modified":"2025-09-23T01:54:09","modified_gmt":"2025-09-23T01:54:09","slug":"scale-ai-launches-seal-showdown-an-alternative-to-lmarena-leaderboard","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/nz\/37731\/","title":{"rendered":"Scale AI launches Seal Showdown, an alternative to LMArena leaderboard"},"content":{"rendered":"

In the years since OpenAI launched ChatGPT to the world<\/a>, kicking off the generative AI boom, developers have relied on LMArena (previously Chatbot Arena) as the default AI leaderboard. Now, Scale AI is bringing some much-needed competition to the AI benchmarking space with its new Seal Showdown benchmarking tool.<\/p>\n

Like LMAerna, Seal Showdown allows users to test various AI models head-to-head and vote on which one performs better. However, Scale AI says that unlike LMAerna, Seal Showdown will more closely reflect how everyday users feel about various models. In an X post, Scale CEO Jason Droege said that Seal Showdown “actually captures real preferences, powered by a platform used by real people.” <\/p>\n


\n This Tweet is currently unavailable. It might be loading or has been removed.
\n <\/a><\/p>\n

“Most benchmarks rely on synthetic tests (coding puzzles, math problems) or feedback from a small slice of people,” said Scale AI\u2019s head of product, Janie Gu, in a blog post<\/a>. “They miss the full spectrum of how real people actually use models in their daily lives. By treating diverse users as a monolith and lumping all feedback into one generalized score, critical nuance is lost.”<\/p>\n

Scale AI launched its Safety, Evaluations, and Alignment Lab (SEAL) leaderboards<\/a> last year, but these leaderboards relied on expert evaluations. Now, ScaleAI will offer leaderboards based on user testing, offering an alternative to the LMArena.<\/p>\n

The startup says its new benchmarking tool<\/a> is based on real-world use and feedback from “users spanning over 100 countries, 70 languages, and 200 professional domains.” (The company also provided the precise methodology for Seal Showdown<\/a>.)<\/p>\n

\n Mashable Light Speed\n <\/p>\n

“Showdown introduces something never before seen in public leaderboards: rich user segmentation,” Gu wrote in the blog post announcing the project. “Because rankings are derived from conversations that contributors have on Scale\u2019s Outlier<\/a> platform, Scale is able to verify each user\u2019s country, education level, profession, language, and age \u2014 enabling anyone to see how models perform for people like them.”<\/p>\n

Because of this demographic information, Scale AI will be able to show which models are most popular according to specific regions, languages, ages, or use cases.<\/p>\n

The criticism that Scale AI has with existing leaderboards is that they \u201crely heavily on hobbyist participation\u201d and that current rankings are \u201cbased on a narrow group of users and their interests,\u201d which leads to a misrepresentation of how those LLMs perform in general use. <\/p>\n

LMArena has also been criticized for bias against open models. Critics say that LMArena’s system favors frontier models from big AI companies like Google, xAI, and OpenAI. However, Scale AI’s solution may not be ideal, either. The initial leaderboard results overwhelmingly rank GPT-5 the highest, which may merely reflect user preference rather than objective performance.<\/p>\n

The updated SEAL leaderboards are live now<\/a>. Currently, GPT-5 tops all of the benchmark categories, a stark contrast to LMArena<\/a>, where Google’s Gemini 2.5 Pro, 2.5 Flash, and Veo 3 lead most of the leaderboard categories.\u00a0<\/p>\n

Disclosure: Ziff Davis, Mashable\u2019s parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.<\/p>\n

\n Topics
\n Artificial Intelligence<\/a>
\n OpenAI<\/a>\n <\/p>\n