{"id":755209,"date":"2026-08-31T03:58:22","date_gmt":"2026-08-31T03:58:22","guid":{"rendered":"https:\/\/www.newsbeep.com\/uk\/755209\/"},"modified":"2026-08-31T03:58:22","modified_gmt":"2026-08-31T03:58:22","slug":"the-50-percent-problem-by-benjamin-riley","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/uk\/755209\/","title":{"rendered":"The 50 Percent Problem – by Benjamin Riley"},"content":{"rendered":"

In 2024, Laurence Holt of the XQ Institute published an essay<\/a> titled \u201cThe 5 Percent Problem,\u201d a term that quickly became common parlance within the ed-tech zeitgeist. Holt argued that when it comes to measuring the student-learning outcomes of various ed-tech programs, the results often only apply to the five percent of students who \u201cused the program as intended. The other 95 percent see minimal gains, if any.\u201d<\/p>\n

Holt\u2019s primary focus was on online math programs, and the central concern running throughout his essay involves whether and how students might \u201cget the recommended dosage\u201d of using various ed-tech tools. Whether the problem arises from unmotivated students or unenthused teachers or disparate access to tech or some combination thereof, the challenge when framed this way is getting students to opt in to using a particular technology.<\/p>\n

Generative AI does not suffer from this problem. Uniquely perhaps in the history of ed-tech, AI has been broadly<\/a> embraced<\/a> by students<\/a> worldwide<\/a>, so much so that the phrase \u201cAI is inevitable\u201d has become commonplace in the education discourse. There are pockets<\/a> of<\/a> resistance<\/a>, of course, and over the last six months we\u2019ve seen mounting opposition to AI across multiple vectors<\/a>, most prominently with data centers<\/a>. Nonetheless, we all know that students are using AI frequently. \u201cDosage\u201d is not an issue.<\/p>\n

In the nearly four years of time that\u2019s passed since ChatGPT was commercially deployed, however, we have suffered from a lack<\/a> of high-quality empirical research on the impact of students using generative AI. There are notable exceptions\u2014such<\/a> as<\/a> these<\/a>\u2014but a recent research landscape analysis<\/a> out of Stanford indicated that, out of more than 800 studies of AI in education, a mere 20 employed true causal measures. Meanwhile, at least one prominent meta-analysis<\/a> purporting to show massive learning gains stemming from AI has been retracted, though not before being viewed 400,000 times. The field is a mess<\/a>.<\/p>\n

Perhaps this is because studying AI in \u201cthe real world\u201d poses serious challenges. We know that AI tools are free and widely available to students. We also know that many are using them after school hours for various purposes, including to help with their schoolwork. But quantifying this is very difficult, because it\u2019s hard to peer into the home lives of children. And it\u2019s equally if not more challenging to connect out-of-school behavior to measurable learning outcomes, the data sets are not easy to link. Accordingly, research on AI\u2019s education impact has tended toward lab studies or small-scale evaluations.<\/p>\n

No longer.<\/p>\n

Last week, a study<\/a> titled \u201cThe Generative AI Learning Penalty: Evidence from Chinese Secondary Education,\u201d authored by David Str\u00f6mberg, Victor Lei, and Yanhui Wu, went viral after The Economist published<\/a> data visualizations from the research paper. The preprint came out in June\u2014not sure how I missed it\u2014and uses data from approximately 27,000 Chinese students in grades seven to 12 to explore a straightforward question that has preoccupied me for several years: \u201cHow does self-directed use of generative AI affect cumulative learning over time in ordinary school settings?\u201d<\/p>\n

We will get into the details shortly, but here\u2019s the headline summary of the results:<\/p>\n

Between 2023 and 2025, approximately 80 percent of all students started using generative AI, and after just five months of use, approximately 50 percent of these students engaged in \u201cfull homework outsourcing,\u201d meaning, they essentially stopped doing their homework independently and used AI instead.<\/p>\n

Over time, this caused significant learning loss as measured both on monthly closed-book exams and via the comprehensive high-stakes exams that China uses to determine high school and college placement for students. On those tests, by the end of June 2025, overall student performance fell by 24% on the former and 18% on the latter, a massive decline.<\/p>\n

What\u2019s more, there is no evidence\u2014none\u2014of any corresponding learning benefit arising from students using AI.<\/p>\n

As such, the researchers state plainly that \u201cour findings show that generative AI, which is likely to become a prevalent technology for education, has a substantial negative impact on student learning.\u201d (My emphasis)<\/p>\n

Put simply, we now have rigorous, empirical evidence on the real world, long-term impact of students using AI. This research indicates that AI substantially harms learning for half of all students, what the researchers call the \u201cGenerative AI penalty,\u201d and provides no discernible benefit to the other half. Again, by the time this study concluded, 50 percent of students using AI with regularity had fully outsourced their homework to generative AI. For these students, when at home, they just stopped thinking about their schoolwork.<\/p>\n

I hereby dub this the 50 Percent Problem. Unlike the Five Percent Problem, the 50 Percent Problem is a measure of harm stemming from widespread adoption of AI by students, rather than their failure to use this technology. It is my conservative estimate of how many students are being directly harmed by using these tools. As I\u2019ve said<\/a> repeatedly<\/a> for several years, generative AI is a tool of cognitive automation. It\u2019s both predictable and tragic that students are learning less because of it. <\/p>\n

This new empirical research provides us with a clear measure of the degree of AI\u2019s educational harm.<\/p>\n

Let\u2019s turn to the data itself. In what follows, we\u2019ll be looking at the performance of three different groups of students across three different learning activities.<\/p>\n

As to the students, we first have the baseline group of student results from the period prior to the introduction of generative AI\u2014these students are labelled \u201cPre AI,\u201d and charted in green. Next, we have the admirable group of students, all 19 percent of them, who did not use generative AI at any point between 2023 and June 2025 when the study concluded\u2014these are labelled \u201cNever AI,\u201d and charted in blue. Finally, we have the remaining 81 percent of students who adopted AI and used it for their schoolwork\u2014these are labelled \u201cUsed AI,\u201d and charted in red. Of note, these students did not all start using AI at the same time, adoption phased in over time (which allowed the researchers to uncover some interesting things that we won\u2019t go into here).<\/p>\n

As to the learning activities, we\u2019ll start by looking at the amount of time that students spent on their homework; the researchers were able to track this because students had to log in online to download their homework and upload it upon completion. Then, we\u2019ll look at how students scored on this same homework. Last but definitely not least, we\u2019ll examine how student performance changed on monthly closed-book exams, as well as China\u2019s very high-stakes high school and college entrance exams. <\/p>\n

Both the underlying research paper and The Economist present a series of helpful data visualizations of this, but without the underlying raw numbers. So I contacted the researchers who conducted the study and David Str\u00f6mberg (the lead author) graciously agreed to provide me with the underlying histogram data I used to create the three animated GIFs you\u2019re about to see.<\/p>\n

With that as backdrop, we can explore three specific questions.<\/p>\n

\"\"<\/a><\/p>\n

The results here are unsurprising\u2014AI significantly reduces the time students spend doing homework. On average, both the Pre AI and Never AI groups spent about an hour doing it, across a range of 50 to 80 minutes. In contrast, within the Used AI group, most students spent around 45 minutes, and some even cruised through in 25, presumably the minimum amount of time it takes to cut-and-paste answers out of a chatbot.<\/p>\n

This is AI as tool of cognitive automation working exactly as intended.<\/p>\n

\"\"<\/a><\/p>\n

Here again we see that Pre AI students and Never AI students have near-identical results, as we\u2019d expect. But not so with the Used AI students\u2014now, we see a huge shift to the right in purportedly \u201cpositive\u201d outcomes, with scores far higher than even the most studious Never AI student managed to achieve. (Note the scores here were normalized to make the average score a 100 (not the maximum), so a score of 130 means \u201c30 percent above the average,\u201d essentially.)<\/p>\n

To restate, the Used AI students studied less yet scored better than their peers. That\u2019s a pretty sweet deal for them, but it\u2019s worth reflecting on the broader implications of this within schools. When I talk to students about AI, one thing I hear them say frequently is that they don\u2019t want to be played for chumps (my term, not theirs). Meaning, if they know their classmates are using AI and getting better grades as a result, even those inclined to resist AI may feel trapped into using it, just to keep pace.<\/p>\n

It\u2019s a cognitive race to the bottom.<\/p>\n

I\u2019ll also add that homework scores are obviously a very imperfect measure of student learning. In my view, a great deal of education research, and certainly the studies often promoted by ed-tech vendors, use comparable \u201cpoint in time\u201d data of supposed learning akin to what we see here. This is understandable, to a degree\u2014it\u2019s very difficult to information on long-term learning outcomes. But what ultimately matters in education is building durable student knowledge. The seductive danger of AI exposed here is that, by using AI, students may falsely have believed everything was proceeding swimmingly. In this sense, AI fosters a mental masquerade, scores go up as actual learning goes down.<\/p>\n

Now to remove the mask.<\/p>\n

\"\"<\/a><\/p>\n

Here is where the proverbial rubber meets the road.<\/p>\n

Consistent with the two previous data sets, we again see near-perfect alignment between the Pre AI and Never AI students on the monthly closed-book exams administered to students. But now the adverse impact of AI is laid bare\u2014just look at that shift to the left. The Used AI students are scoring at levels far lower than their Never AI peers, indeed, many score far lower than anything recorded prior to AI existing. <\/p>\n

I\u2019ve labeled this the Deadweight Learning Loss to underscore the volume of harm caused by the generative AI learning penalty. What\u2019s we\u2019re seeing here is unambiguous evidence of a decline in overall student performance that grew over time as students adopted AI. In fact, the average decline was 20 percent with a 1.4 standard deviation (SD). Although it\u2019s far from an apples-to-apples comparison, this vastly exceeds the estimated learning loss<\/a> in the US after the pandemic (approximately .25 SD in math and .13 SD in reading). <\/p>\n

What\u2019s more, over time this Deadweight Learning Loss had significant adverse consequences for students on the comprehensive high-stakes high school and college entrance exams that China administers to determine student placement. For students who adopted generative AI two or more years prior to being tested, the estimated negative effects are 24 percent (1.5 SD) for the high school exam, and 18 percent (1.3 SD) for the college exam. <\/p>\n

If you are a parent with a child in junior high or high school who uses generative AI, this data should terrify you. This isn\u2019t about academic integrity, it\u2019s about a generation of kids being told AI is \u201cinevitable\u201d and \u201cthe future\u201d and acting accordingly, in a societies where we\u2019ve yet to develop firm norms about what\u2019s acceptable to do with these tools within education. The upshot is that students are using AI in ways that are harming their cognitive development and foreclosing their life opportunities. Harms that are not easily remediated, if at all.<\/p>\n

This is the 50 percent problem. Generative AI is cognitive cancer and we are doing next to nothing to stop it from spreading.<\/p>\n

I\u2019ll now address a few objections.<\/p>\n

First, I want to credit The Economist for putting this research on my radar, and for raising broader attention to its harrowing findings. Yet, remarkably, in the brief article accompanying the research data we\u2019ve just covered, the anonymous magazine author suggests that \u201cAI can boost learning productivity but only for those who use the technology intelligently.\u201d In similar fashion, Blake Richards, a researcher at Google who works \u201con the intersection of machine learning and neuroscience,\u201d argued<\/a> on social media that AI is not really the problem here, because \u201cif you control for how long students spend studying, then students using AI actually perform equal or better\u201d than those that did not.<\/p>\n

Motivated reasoning is a powerful force. Here, of course, \u201ccontrolling\u201d for how long students spend studying erases the major findings of this research. But let\u2019s leave that aside, and probe whether this argument can be justified on its own terms\u2014does this study suggest that AI can boost learning if used \u201cintelligently\u201d?<\/p>\n

As best I can tell, this claim is premised on this graph that correlates homework completion time with exam results:<\/p>\n

\"\"<\/a><\/p>\n

If we ignore all that pesky data on the left side of this chart (which of course we shouldn\u2019t), and just focus on the overlap between the Generative AI students who continued to study for the same duration as the No Generative AI students, it\u2019s true there\u2019s no major gap between them. Of course, note that there\u2019s no discernible benefit to using AI either\u2014apart that is from the spike at the very top end.<\/p>\n

So what\u2019s the deal with the spike? Might we cling to it as proof of the \u201cpromise\u201d of AI? Well, here\u2019s a good lesson on why one should always be careful eyeballing charts without access to the relevant underlying data, because when I asked Str\u00f6mberg how many Used AI students persisted in studying for at least 75 minutes, he told me this (via email, with my emphasis):<\/p>\n

It is very rare for students who have adopted AI to spend 75 minutes on homework. This occurs for only 20 students, and for each of them only in a single month (0.2% of AI student-month observations). More than five months after AI adoption, we never observe students spending 75 minutes on homework. In fact, in this group, only four students spend more than 65 minutes on homework.<\/p>\n

Let\u2019s get real. Given this study involved almost 27,000 students, indexing on this vanishingly small number of studious AI-using students isn\u2019t just putting lipstick on a pig, it\u2019s smearing its body in Revlon from snout to tail. We need to stop pretending there is some massive benefit to learning if only we trained kids properly on how to use AI. It\u2019s fundamentally harmful.<\/p>\n

A different and more sophisticated counterargument might proceed along the following lines: This study tracked student usage of AI stemming from its earliest days, when the tools were less capable than they are today. Perhaps relatedly, the researchers here found evidence that it was the early AI adopters most harmed by generative AI, insofar as \u201cthe estimated AI learning penalty fell from around 25 percent in early 2023 to around 16 percent by June 2025.\u201d As such, perhaps this study constitutes the \u201chigh-water mark\u201d of AI-induced harm, and perhaps the AI learning penalty will continue to drop. Or so we might hope, anyway.<\/p>\n

But as the cliche goes, hope is not a strategy, and there are some problems with this counterclaim. For one thing, the rapid evolution of AI cuts both ways\u2014we now have AI companies explicitly marketing \u201cAI agents\u201d to students<\/a> to complete their tasks for them. Agents are even worse than chatbots from a cognitive development standpoint\u2014at least the latter require an interaction of some sort to produce output. For another, even if the magnitude of the learning penalty continues to diminish, there\u2019s a question of volume, too. Recall that 20 percent of students managed to resist using AI prior to June 2025. Do you think that number has gone up or down since then? I know my bet.<\/p>\n

Finally, I can imagine someone saying I\u2019m placing too much weight on this research\u2014it\u2019s just one study from China, after all. On that front, and as a self-sanity check, I asked three PhD education researchers to review the methodology employed, and all three came back with positive reviews (\u201cit\u2019s very good work,\u201d said one). And China is surely the one country in the world that can match the US for AI adoption and enthusiasm, though it may surprise you to learn that AI companies in China disable their products<\/a> completely during the high-stakes testing periods. (OpenAI, in revealing contrast, heavily promotes<\/a> ChatGPT on college campuses during finals week in the US.) <\/p>\n

Perhaps more importantly, if you are an AI-in-Education Enthusiast, I feel confident in saying that you will not be able to produce research of comparable rigor that shows positive long-term education impact of generative AI in real-world conditions comparable to those here. If there were such evidence, my inbox would be filled with people jamming it down my throat, trust me. And no, this single study<\/a> from 2024 involving roughly 150 physics students at Harvard is not going to cut it, sorry.<\/p>\n

The 50 Percent Problem will not disappear of its own accord. The Edu-Cognoscenti continues to chatter<\/a> about learning loss related to school closures during the pandemic. Well, the learning loss stemming from generative AI appears more substantial, it\u2019s happening right now, and it may endure for far longer. So to all the policymakers and philanthropists and so-called thought leaders who purport to take education evidence seriously, I ask you: What are you doing to prevent ongoing educational harms of generative AI? Are you doing anything at all?<\/p>\n

We are in cognitive crisis.<\/p>\n

I\u2019ll close with this. As I was drafting this essay, my friend Dan Willingham published his perspective<\/a> about when students should use AI. Please read it. Although he\u2019s a tad more measured (or perhaps just realistic) about its role in education, his core contention echoes what I\u2019ve been arguing for several years, and stems from a basic understanding of human cognition:<\/p>\n

[T]he point of assignments is the mental processes required to complete them, and the point of the mental processes is learning. That seems to suggest a simple litmus test for the use of AI. Artificial Intelligence tools should not substitute for tasks wherein students would benefit from doing the mental work themselves. Only use AI for what you already know how to do.<\/p>\n

A tool that should only be used once you already know something is not a learning tool. We must stop gesturing at AI\u2019s imagined potential, and focus our efforts instead on mitigating the 50 Percent Problem it\u2019s created.<\/p>\n

How, you might reasonably ask? It won\u2019t be easy. But I have some emerging ideas, stemming from recent investigations into communities of technological refusal. It\u2019s been quite the learning journey. More soon.<\/p>\n

My thanks to David Str\u00f6mberg for providing his underlying data and to my anonymous academic friends who reviewed the study and this essay\u2014all errors mine and mine alone, of course. <\/p>\n","protected":false},"excerpt":{"rendered":"In 2024, Laurence Holt of the XQ Institute published an essay titled \u201cThe 5 Percent Problem,\u201d a term…\n","protected":false},"author":2,"featured_media":755210,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[59,90,56,54,55],"class_list":["post-755209","post","type-post","status-publish","format-standard","has-post-thumbnail","category-science","tag-gb","tag-science","tag-uk","tag-united-kingdom","tag-unitedkingdom"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/755209","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=755209"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/755209\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/755210"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=755209"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=755209"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=755209"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}