He noted that ARC Prize has had “a lot of back and forth with OpenAI about how to best test their models, especially with regard to compaction,” and welcomed the company “starting to figure out the answer.” Different providers using different settings does create “a potential parity issue,” Chollet said, but he considers that acceptable “as long as the settings and the cost are clearly reported.”<\/p>\n
Original article:<\/p>\n
OpenAI says it can keep up on ARC-AGI-3.\u00a0After Anthropic’s\u00a0Claude Opus 5 quadrupled the record score on the logic benchmark<\/a>,\u00a0OpenAI is now showing<\/a>\u00a0that GPT-5.6 Sol hits 38.3 percent with two API settings, beating Opus 5’s 30.2 percent.<\/p>\n OpenAI isn’t using the official test environment, though. Instead, it runs GPT-5.6 Sol through its own Responses API<\/a> with “Retained Reasoning,” which keeps the model’s chain of thought between steps, and\u00a0“Compaction,”\u00a0which summarizes old context instead of truncating it. In the official harness, GPT-5.6 Sol scored just 7.8 percent because the model’s reasoning gets discarded after each action.<\/p>\n OpenAI argues that benchmarks never measure just the model but also the technical setup around it. That’s true, and\u00a0ARC-AGI-3\u00a0is designed<\/a> to test pure model performance. The official ARC scores use a standardized approach without provider-specific settings to ensure fair comparisons,\u00a0ARC Prize said in response to OpenAI’s results<\/a>. The sticking point is whether ARC Prize used an older “OpenAI-style completions API” that lacked features the Claude API already offered, which would make the comparison unfair to OpenAI.<\/p>\n \t\t\t\tAI News Without the Hype \u2013 Curated by Humans<\/p>\n \n\t\t\t\t\tSubscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive “AI Radar” frontier report six times a year, full archive access, and access to our comment section.\t\t\t\t<\/p>\n![\"\"](\"https:\/\/www.newsbeep.com\/il\/wp-content\/uploads\/2026\/07\/GPT-5.6-Sol-on-the-ARC-AGI-3-Public-Set.png\")
<\/a>GPT-5.6 Sol’s ARC-AGI-3 scores jump dramatically when using OpenAI’s custom harness with retained reasoning and compaction, compared to the official test harness. | Image: OpenAI<\/p>\n