Suketu Patel<\/a> is a doctoral candidate in comparative and cognitive psychology at the Graduate Center of the City University of New York. Patel and his colleagues conducted this research in the laboratory of Jin Fan at Queens College, CUNY. He noted that the initial public reception of modern language models inspired the research team to investigate the software\u2019s true cognitive capabilities.<\/p>\n\u201cWhen ChatGPT arrived, much of the excitement centered on its capacity for task completion, theory of mind, and emotional intelligence,\u201d Patel said. \u201cYet it was also prone to hallucination and confabulation. LLM performance was strong on some tasks and surprisingly weak on others. We wanted a canonical attention task to rigorously probe these systems and compare them to biological attention.\u201d<\/p>\n
Human attention is a complex process supported by multiple interconnected brain networks. \u201cThe Stroop task is fitting because the success of LLMs rests on the transformer\u2019s attention mechanism,\u201d Patel said. \u201cIn humans, attention comprises three dissociable yet overlapping systems: alerting, orienting, and executive control. So we set out to test whether these models possess all three.\u201d<\/p>\n
The Stroop task, first introduced in the 1930s, measures how well a subject handles conflicting information. In a standard version, a participant might see the word \u201cBLUE\u201d printed in red ink, and they must name the ink color instead of reading the text. \u201cIt is worth emphasizing that the Stroop task is not a test of thinking or higher-order reasoning,\u201d Patel said. \u201cIt specifically targets conflict resolution and inhibition.\u201d<\/p>\n
The automatic human response is to simply read the word itself, which requires active mental suppression to overcome. \u201cThe core idea is that word reading is essentially automatic in humans, a heavily trained prior that becomes what we call a prepotent response, the one that fires first and strongest,\u201d Patel explained. \u201cAI is in a similar position, since it is far more trained to read words than to name colors.\u201d<\/p>\n
The researchers examined two leading artificial intelligence models: OpenAI\u2019s GPT-4o and Anthropic\u2019s Claude 3.5 Sonnet. The models received an image prompt and were asked to either read the text of the words presented or name the physical color of the text. The team tested the programs using five different conditions, including words printed in matching colors, non-matching colors, a mixed condition, neutral office words, and strings of the letter \u201cX.\u201d<\/p>\n
To test how well the programs could sustain their attention, the scientists varied the number of words presented in each image, ranging from one to forty words. \u201cGoal maintenance is the ability to hold onto an instruction and keep following it in any context while filtering out interfering information,\u201d Patel said. \u201cHumans develop this capacity over time. AI can certainly follow instructions and reach goals, but it does so in a fundamentally different way, and that difference becomes more visible as the context grows longer or contains conflicting information.\u201d<\/p>\n
When processing short lists of one or five words, the artificial intelligence models performed much like humans. They achieved high accuracy on the word-reading task and showed a slight dip in performance during the mismatched color-naming trials. However, as the lists grew longer, the performance of both models on the incongruent condition collapsed completely.<\/p>\n
GPT-4o accurately named the ink color on incongruent trials 91 percent of the time with five-word lists. This accuracy plummeted to just 1 percent on both the twenty-word and forty-word lists. Claude 3.5 Sonnet maintained stability slightly longer but eventually dropped to just 10 percent accuracy on the forty-word incongruent lists.<\/p>\n
During these failures, the models entirely abandoned the instruction to name the color and defaulted back to reading the text. \u201cWe were surprised by how accuracy broke down at relatively small context sizes, with lists as short as 10 words,\u201d Patel said. \u201cWhat made this striking was the contrast with the nonword conditions, i.e., XXXX, where accuracy was nearly perfect. That gap highlights just how automatic reading behavior in LLMs, like in humans, also requires meaningful words.\u201d<\/p>\n
The researchers suggest that artificial models experience this breakdown because their programming lacks the forceful oversight found in the human brain. \u201cOur central argument is that the limitation stems from the lack of an explicit mechanism for top-down modulation,\u201d Patel told PsyPost. \u201cThis is when a rule or goal enforces priority among competing representations from the outset, proactively, and can sustain a constraint by inhibiting a prepotent prior rather than down-weighting.\u201d<\/p>\n
Without this mental override, the models are overwhelmed by their basic programming habits. \u201cThe study shows that, at the signal level, the ability to detect and resolve the conflict degrades because transformer attention can only impose a soft constraint on that automatic reading, rather than the hard one that an executive control mechanism would provide,\u201d Patel added.<\/p>\n
Newer artificial intelligence systems sometimes attempt to bypass this problem using added programming layers. \u201cScaffolding methods we see in the latest AI systems have tool use, thinking, and code generation to stand in for that missing component, but each is bolted onto a base model that still propagates errors,\u201d Patel said.<\/p>\n
Relying on outside code to solve the test fundamentally misses the point of the cognitive assessment. \u201cThis is why any strategy that avoids suppressing prepotent word reading defeats the purpose of the Stroop task,\u201d Patel explained. \u201cA few of the models we studied are inconsistent about whether they reach for code, but when they do run code, they tend to solve the task perfectly.\u201d<\/p>\n
The scientists address this issue extensively in their report, noting that relying on code generation is not true cognitive control. \u201cShortcutting the task through chain-of-thought reasoning or code generation is really just avoiding it, papering over a deficiency at the signal level that becomes critical as goals grow more complex,\u201d Patel said. \u201cHumans can cheat in exactly the same way. We can verbalize the answer, blur our vision, or use a tool to keep from reading the word, and each of those moves invalidates the assessment.\u201d<\/p>\n
The study does carry certain methodological constraints, and the researchers note that models might eventually pass similar tests through brute-force pattern recognition. \u201cWe are not claiming that LLMs cannot do this task,\u201d Patel said. \u201cWith more training data, they could likely handle even larger contexts reliably.\u201d<\/p>\n
\u201cBut that would be a task-specific kind of gating, achieved through sheer exposure, rather than the general form of control that does not depend on heavy training,\u201d Patel added. \u201cIt is also worth noting that few tasks share the Stroop task\u2019s particular dynamic, in which one response (reading) is so strongly pre-activated that it competes with the instructed response (naming the color).\u201d<\/p>\n
These findings present a challenge to current assumptions within the technology industry. \u201cSo the Stroop task is diagnostic of a structural constraint in LLMs, not simply a measure of task performance,\u201d Patel said. \u201cThe bitter lesson, and the implicit wager behind scaling to larger models toward artificial superintelligence (ASI), is that this gating mechanism, what neuroscience calls executive control, will emerge from more scale and data without any dedicated architecture.\u201d<\/p>\n
Future development in artificial intelligence may need to move beyond simply increasing data processing speeds or expanding text databases. \u201cWe have begun exploring how executive control could be built directly into current AI architecture,\u201d Patel said. \u201cWe see it as an essential ingredient for long-horizon instruction following, the ability to stay on task across extended and complex interactions.\u201d<\/p>\n
The study, \u201cDeficient executive control in transformer attention<\/a>,\u201d was authored by Suketu Chandrakant Patel, Hongbin Wang, and Jin Fan.<\/p>\n","protected":false},"excerpt":{"rendered":"New research provides evidence that while advanced artificial intelligence models process language with remarkable skill, they struggle significantly…\n","protected":false},"author":2,"featured_media":654962,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[59,90,56,54,55],"class_list":["post-654961","post","type-post","status-publish","format-standard","has-post-thumbnail","category-science","tag-gb","tag-science","tag-uk","tag-united-kingdom","tag-unitedkingdom"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/654961","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=654961"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/654961\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/654962"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=654961"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=654961"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=654961"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}