Using these tools requires understanding human performance. How do real people, our only proof point of general intelligence, actually learn and solve novel problems?<\/p>\n
Today we\u2019re releasing the human dataset for ARC-AGI-3<\/a> – a controlled study of 458 participants and the most exhaustive human testing study in the ARC-AGI series to date.<\/p>\n We do not yet have AGI. This dataset is the receipt.<\/p>\n ARC-AGI-3 Public Demo environments: AR25<\/a>, LF52<\/a>, SB26<\/a> ARC-AGI-3<\/a> is a series of 135 abstract reasoning environments. Play them yourself<\/a>.<\/p>\n The test taker, whether human or AI, is not given instructions on how to play. They must explore, infer the rules that govern the environment, and come up with a strategy on their own.<\/p>\n A key design constraint of ARC-AGI-3 is that every environment must be solvable by humans with no prior training. To ensure this, we conducted the largest formal study on ARC-AGI human performance ever done.<\/p>\n To dive deeper into ARC-AGI-3, check out the benchmark<\/a>, play the environments<\/a>, or watch our launch video<\/a>.<\/p>\n Human Baselines on ARC-AGI-3<\/p>\n To gather human baselines we tested members of the general population. Participants included various levels of education, income, job sectors, and ages. We did not control for one particular demographic.<\/p>\n We held weekly, in-person, focus groups in a San Francisco-based testing center. No references to ARC Prize Foundation or AI testing were made at any time.<\/p>\n Each testing session lasted 90 minutes. Every participant received a base payment of ~$130, plus an extra $5 for each environment they successfully solved.<\/p>\n Tests were held under “first-run” conditions. This means every participant only saw environments once and had a single attempt to beat it. This measures the ability to learn and adapt to a novel problem, not the ability to repeat a previously learned solution.<\/p>\n Humans were given the same prior information and affordances that AI is given:<\/p>\n Both humans and AI get the same “system prompt\u201d Humans and AI received identical information. Neither had an informational advantage.<\/p>\n As testing progressed, the data we collect shows us what intelligence looks like in practice.<\/p>\n If an environment is too difficult, it was excluded from the dataset or revised.<\/p>\n The end result is a dataset that demonstrates human solvability of 100% of environments. Every environment is beaten by at least two independent participants and most are beaten by many more.<\/p>\n ARC-AGI-3 Human Testing Dataset<\/p>\n We’re open sourcing the full Public Demo dataset<\/a>, which includes 342 human step-by-step replays for our 25 public environments. With it researchers can analyze:<\/p>\n Per-environment solvability Solvability<\/p>\n The most basic measure of human performance is solvability: can humans complete these environments? To answer that, we look at completion rates.<\/p>\n Our early findings:<\/p>\n Not all environments are solved equally – 10 out of 10 participants solved r11l<\/a>, whereas only 6 out of 12 solved tr87<\/a> The table below summarizes each of the 25 public demo environments, including how many participants attempted each environment, how many solved it, and links to the top 10 replays (scores = levels completed).<\/p>\n Env.PlaysSolvesTop 10 Replays (score)ar25<\/a>1058<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 7<\/a>, 6<\/a>, 6<\/a>, 5<\/a>, 0<\/a>bp35<\/a>1429<\/a>, 9<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 6<\/a>cd82<\/a>1186<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 2<\/a>, 2<\/a>cn04<\/a>1266<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 3<\/a>, 3<\/a>, 2<\/a>, 1<\/a>dc22<\/a>1146<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 5<\/a>, 5<\/a>, 4<\/a>, 4<\/a>, 4<\/a>, 3<\/a>ft09<\/a>1046<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 3<\/a>, 2<\/a>, 2<\/a>, 1<\/a>, 0<\/a>, 0<\/a>g50t<\/a>1437<\/a>, 7<\/a>, 7<\/a>, 5<\/a>, 3<\/a>, 3<\/a>, 2<\/a>, 2<\/a>, 2<\/a>, 2<\/a>ka59<\/a>1037<\/a>, 7<\/a>, 7<\/a>, 6<\/a>, 5<\/a>, 1<\/a>, 1<\/a>, 0<\/a>, 0<\/a>, 0<\/a>lf52<\/a>11410<\/a>, 10<\/a>, 10<\/a>, 10<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 1<\/a>, 1<\/a>lp85<\/a>54158<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>ls20<\/a>1367<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 5<\/a>, 5<\/a>, 2<\/a>, 1<\/a>m0r0<\/a>1176<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 5<\/a>, 4<\/a>, 1<\/a>r11l<\/a>10106<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>Env.PlaysSolvesTop 10 Replays (score)re86<\/a>1158<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 2<\/a>, 2<\/a>s5i5<\/a>1148<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 7<\/a>, 4<\/a>, 4<\/a>, 2<\/a>, 2<\/a>, 2<\/a>sb26<\/a>1258<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 7<\/a>, 6<\/a>, 4<\/a>, 4<\/a>, 4<\/a>sc25<\/a>15106<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>sk48<\/a>1478<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 7<\/a>, 4<\/a>, 4<\/a>sp80<\/a>1226<\/a>, 6<\/a>, 5<\/a>, 4<\/a>, 4<\/a>, 2<\/a>, 2<\/a>, 1<\/a>, 1<\/a>, 1<\/a>su15<\/a>1339<\/a>, 9<\/a>, 9<\/a>, 4<\/a>, 4<\/a>, 4<\/a>, 3<\/a>, 3<\/a>, 1<\/a>, 1<\/a>tn36<\/a>1467<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 1<\/a>tr87<\/a>1266<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 6<\/a>, 5<\/a>, 5<\/a>, 5<\/a>, 4<\/a>tu93<\/a>1399<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 4<\/a>vc33<\/a>1067<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 7<\/a>, 6<\/a>, 3<\/a>, 2<\/a>, 0<\/a>wa30<\/a>1459<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 9<\/a>, 8<\/a>, 8<\/a>, 8<\/a>, 6<\/a>, 5<\/a>Public Demo Set 25 Environments: 342 plays, 145 solves. Note: lp85 has higher play counts due to early inclusion as an ARC-AGI-3 preview environment in July 2025. How humans beat the environments matters too. Efficiency measures not just whether a level is completed, but how many actions it took relative to the human baseline. This connects directly to the definition of intelligence<\/a>, skill-acquisition efficiency.<\/p>\n ARC-AGI-3 offers the first formal measure of learning efficiency in the ARC-AGI series. The chart below shows the distribution of human per-level efficiency across the public demo set.<\/p>\n Since previewing ARC-AGI-3, nearly one million scorecards have been submitted on public environments. That real-world data helps us stress-test and harden our scoring approach.<\/p>\n What we\u2019ve observed:<\/p>\n “Luck factor”: In at least one level of one game, the environment restricted determining the optimal path from the start. An early choice could lock a player out of the minimum action count regardless of how well they played afterward. Imperfect information isn’t inherently a problem, it’s a natural feature of many environments. However, it is an issue when the baseline is tight enough that one unlucky early choice dominates a player’s score. Our original human baseline (2nd best human) forced luck into what should be a pure measure of reasoning efficiency. Based on what we\u2019ve observed, we\u2019re announcing two updates to ARC-AGI-3 scoring:<\/p>\n The per-level baseline is now less sensitive to outlier performances, reducing the impact of luck on individual levels. The net result of these changes is a marginal increase in scores for both humans and AI (+0.5pp) and better reflects our desire to fairly compare efficiency between test taskers.<\/p>\n Our core claim for ARC-AGI-3 remains:<\/p>\n ARC-AGI-3 is 100% solvable by humans. Every environment was beaten by at least two humans, typically five or more, out of a small uncontrolled panel of around ten members of the general public.<\/p>\n To see how this works in practice, here’s the action progression chart for re86<\/a> from our 10 human testers.<\/p>\n When AI scores 100% on ARC-AGI-3 it means AI beat every level of every environment at or above the median human-baseline action efficiency.<\/p>\n To read more about our scoring, see our documentation<\/a>.<\/p>\n Looking ahead<\/p>\n We believe our human study for ARC-AGI-3 is the largest of its kind for an AI benchmark and we\u2019re proud to fully open source the dataset to support our mission.<\/p>\n Going forward, we\u2019ll continue to develop benchmarks that are grounded the understanding of human performance.<\/p>\n We\u2019ll continue releasing data, improving methodology, and refining scoring<\/p>\n If you\u2019d like to join us on this journey, consider making a tax-deductible donation<\/a> to ARC Prize Foundation or exploring roles<\/a> on our team.<\/p>\n","protected":false},"excerpt":{"rendered":"AGI is here when a system can learn like a human. However there is still a gap between…\n","protected":false},"author":2,"featured_media":380160,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[365,363,364,111,139,69,145],"class_list":{"0":"post-380159","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-new-zealand","12":"tag-newzealand","13":"tag-nz","14":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/380159","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=380159"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/380159\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/380160"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=380159"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=380159"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=380159"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nARC-AGI-3<\/p>\n
\nTest takers are told available environment actions
\nNeither are told that this was an ARC Prize or ARC-AGI test<\/p>\n![\"Human](\"https:\/\/www.newsbeep.com\/nz\/wp-content\/uploads\/2026\/04\/human_system_prompt.png\")
The instruction modal used during testing. These are the only instructions given to humans.<\/p>\n
\nAction counts and efficiency distributions
\nFull replay data (step-by-step human interactions)<\/p>\n
\nPer-level data reveals where players get stuck – in cd82<\/a>, 2 of 11 players couldn’t get past level 2, but most who did went on to solve the entire environment. This suggests a steep onboarding curve rather than overall difficulty<\/p>\n
\nEfficiency<\/p>\n![\"Distribution](\"https:\/\/arcprize.org\/media\/images\/MedianHumanARCAGI3.svg\")
Each bar represents how many human level-solves fell at that efficiency. 100% means the player solved the level in the same number of actions as the median human. Above 100% = fewer actions; below = more.
\nARC-AGI-3 Scoring Updates<\/p>\n
\nFirst place doesn\u2019t always get 100%: We observed that test takers could beat the human baseline on every single level by a wide margin, but if they were subpar on just one level, the hard cap of 100% per level meant their overall score dropped below 100%. This did not accurately reflect the spirit of measuring a game’s action efficiency.<\/p>\n
A single unusually efficient human run no longer defines the baseline for ARC-AGI-3 scoring. Rather the baseline now reflects more typical human play. Technical change: the human baseline which normalizes scores moves from 2nd-best player to median player per level.
\nA single subpar level no longer disproportionately drags down an overall score
A test taker who generalizes well across an entire environment is no longer penalized by a single, sub-par, level. Technical change: per-level score cap increases from 100% to 115%.<\/p>\n![\"Action](\"https:\/\/arcprize.org\/media\/images\/re86ActionProgression.svg\")
Level progression vs Action chart for re86<\/a>. Notice how five players finish the game, but only four score 100% – they were more efficient than the human baseline. The fifth player finished but scored 89% because they weren’t as efficient as the baseline.<\/p>\n