Sign up to see the future, today<\/p>\n
Can\u2019t-miss innovations from the bleeding edge of science and tech<\/p>\n
Good news for sports broadcasters and fans who\u2019d prefer their play-by-plays to have that human-touch: AI doesn\u2019t know ball.<\/p>\n
A new study by researchers at the University of North Carolina at Chapel Hill and Northeastern University found that top AI models are horrible at analyzing professional sports. The yet-to-be-peer-reviewed study<\/a> sought to analyze how capable the most popular AI models are in the fields of perception, reasoning, simulation, and agency \u2014 four traits which are difficult to evaluate with existing testing methods.<\/p>\n To probe how AI performs in these areas, researchers turned to the wide world of sports to create a new kind of AI test. Called strategic video intelligence, or \u201cSVI-bench,\u201d the novel test comprised 35,000 hours of sports footage from basketball, soccer, and hockey, as well as 15 million annotated plays, 15,000 hours of professional analysis, 23,000 post-game reports, and 103,000 statistical records.<\/p>\n Where AI performed the best was in perception: identifying which player performs which action at a given point in the match. But even there, they struggled badly. The models, which included ChatGPT, Google\u2019s Gemini, and the open-source model Qwen, successfully eyeballed which player was doing what roughly 74 percent of the time \u2014 a rate which would get even a volunteer Little League announcer sacked.<\/p>\n The AI models did far worse on causal reasoning, or explaining why certain plays went down the way they did, with success rates falling near 40 percent on average. For example, when researchers asked the models to identify what was unusual about a Cody Martin three-pointer<\/a> \u2014 which bounced off the top of the backboard before landing in the bucket \u2014 ChatGPT replied that it was \u201chis first made three of the game.\u201d<\/p>\n Simulation, or asking AI to find evidence to predict things like where a player would physically go based on their trajectory, was also dismal. During these tests, the best-performing model was functionally flipping a coin in order to guess a player\u2019s next steps, and performance dropped even further when models were asked to plot out longer motion toward a goal or basket.<\/p>\n As computer science researcher at Northeastern and study co-author Lorenzo Torresani said in a press blurb by the university<\/a>, AI \u201ccannot tell you why things happen, and it cannot tell you what\u2019s gonna happen next.\u201d<\/p>\n When researchers probed the models\u2019 agency \u2014 basically asking them to make complex post-game analysis of stats and trends, like a human broadcaster would \u2014 its accuracy fell to just 5 percent.<\/p>\n \u201cA good sportscaster does much more than describe what\u2019s on screen \u2014 they explain why a play worked, anticipate what\u2019s next, and\u2026 decide which moments matter,\u201d Torresani said. \u201cOur study shows AI is already reasonably good at the descriptive part, but collapses on the rest.\u201d<\/p>\n While sportscasters can definitely breathe a sigh of relief, the study\u2019s findings are also good news for other knowledge workers, at a time when there\u2019s been relentless fear of AI automation turning the job market inside out<\/a>.<\/p>\n \u201cThe same gap shows up in any job whose value lies not in describing what\u2019s visible, but in understanding why events unfold, anticipating what comes next, deciding what matters, and recommending what to do about it,\u201d Torresani concluded.<\/p>\n