Like many human workers, AI agents might struggle to make a living off online gig work.<\/p>\n
For these autonomous AI systems, however, it\u2019s not for lack of work, but because they overwhelmingly fail to complete projects sourced from online freelance platforms.<\/p>\n
That\u2019s what researchers at the Center for AI Safety along with Scale AI found when they tested agents on their ability to perform economically valuable work as part of a new benchmark, the Remote Labor Index (RLI). (The \u201chighest-performing agent\u201d managed a piddly automation rate of just 2.5%.)<\/p>\n
Plenty of existing benchmarks measure specific skills like coding or basic office tasks, and agents have come to master those. But the creators of the RLI wanted to see how those abilities translate into real-world settings where projects might encompass more complex work of greater variety.<\/p>\n
The authors said their findings should serve to ground some of the more provocative claims around AI-fueled automation replacing human workers. Anthropic CEO Dario Amodei, for instance, has predicted<\/a> that AI could wipe out half of all white collar jobs in the next few years.<\/p>\n \u201cCurrently, these agents can\u2019t automate people\u2019s jobs, but AI development moves fast, so things could look very different in five years,\u201d Mantas Mazeika, lead researcher at CAIS, said in an email. \u201cWe hope RLI can help provide clarity here and enable policymakers and the public to proactively navigate AI-driven labor automation.\u201d<\/p>\n The benchmark consists of 240 end-to-end freelance projects across categories like product and graphic design, game development, and architecture, as well as successful examples of completed projects from human professionals. The projects are sourced directly from online freelance platforms.<\/p>\n Even the best agent tested (from the Chinese company Manus) was only able to earn $1,720 of the available $143,991 in freelance payment on offer, followed by Anthropic\u2019s Sonnet 4.5, with $1,280, and GPT-5, \u201cearning\u201d $1,180.<\/p>\n Keep up with the innovative tech transforming business<\/p>\n Tech Brew keeps business leaders up-to-date on the latest innovations, automation advances, policy shifts, and more, so they can make informed decisions about tech.<\/p>\n Agents would submit work in corrupt or empty files, fail to complete pieces of the instructions, or provide shoddy or inconsistent material, according to the paper.<\/p>\n \u201cThere are areas that today\u2019s agents can do really well. Those are\u2026creative tasks like writing a report, solving software problems, or doing some data synthesis integrations,\u201d Bing Liu, Scale AI\u2019s director of research, told Tech Brew. \u201cBut for the real-world tasks that require complex interaction with tools, understanding ambiguous requirements in briefs, and completing [long-running] tasks\u2026those types of tasks still challenge most of these frontier models today.\u201d<\/p>\n The findings contrast sharply with the ongoing hype around agents in the office. Companies like Microsoft<\/a> and Salesforce<\/a> have painted a future where specialized agents tackle most day-to-day work tasks with human employees taking on more of a supervisory role.<\/p>\n Liu said that while these agents might excel at benchmarks devised specifically to test their abilities, real-world use tends to present more varying and complex challenges than what\u2019s in the test. That real-world setting is what the team is trying to capture with this new index.<\/p>\n \u201cSynthetic tasks, most of the time, are created for the sake of challenging models and agents in a certain way for specific capabilities, but real-world tasks are a lot more different than that,\u201d Liu said. \u201cThe implication to business leaders is that we should continue to monitor and measure AI’s progress in these real-world tasks, the real workflows that they truly care about that bring real economic value. And we have a long way there.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"Like many human workers, AI agents might struggle to make a living off online gig work. For these…\n","protected":false},"author":2,"featured_media":272135,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[256,254,255,64,63,105],"class_list":["post-272134","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-au","tag-australia","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/272134","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=272134"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/272134\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/272135"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=272134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=272134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=272134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}