{"id":929944,"date":"2026-10-12T07:04:11","date_gmt":"2026-10-12T07:04:11","guid":{"rendered":"https:\/\/www.newsbeep.com\/ca\/929944\/"},"modified":"2026-10-12T07:04:11","modified_gmt":"2026-10-12T07:04:11","slug":"ai-vulnerability-patching-has-a-grading-problem","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/ca\/929944\/","title":{"rendered":"AI vulnerability patching has a grading problem"},"content":{"rendered":"

Researchers at Northwestern University tested AI coding agents<\/a> on 112 real security bugs. They found that a patch which stops a crash often fails the checks the project\u2019s maintainers wrote for the fix. With Claude Opus 4.7, third of the patches that made the crash go away still failed those maintainer tests.<\/p>\n

\"AI<\/p>\n

If you\u2019re thinking about letting an AI tool patch vulnerabilities for you, this is the number to look at. Many security patching benchmarks grade a fix on one question: does the proof-of-concept input, the sample that triggers the bug, still crash the program? A patch can pass by blocking the path to the crash and leaving the flaw where it was. The authors call that kind of fix \u201cstrictly worse than no fix at all,\u201d since it tells you the problem is solved when it isn\u2019t.<\/p>\n

Two patches, one right answer<\/p>\n

The researchers built<\/a> their dataset from bugs in 84 open-source projects written in C\/C++, Go and Rust. For each bug, they pulled the tests the original developers added alongside the real fix. Then they ran the AI patches against those tests.<\/p>\n

Take a bug in PHP. One AI-style patch fixes the logic the way PHP\u2019s maintainers did. Another just throws an error whenever the inputs look unusual, which stops the crash but also breaks calls PHP is supposed to handle. Both pass PHP\u2019s existing tests. Only the test the maintainers wrote for the fix tells them apart.<\/p>\n

The team also tried three generations of Claude models. The newest stopped the crash about 90% of the time, but it passed the maintainers\u2019 tests only about 61% of the time. So even the best model they tested failed those tests on roughly four in ten bugs.<\/p>\n

The agents found the answers<\/p>\n

The team also caught agents pulling answers out of the test setup:<\/p>\n

Web search: with web search left on, one Claude Code agent fetched the upstream fix commit within its first four tool calls. It later reported that its patch followed the same approach and cited the page it had fetched.
\nGit history: another agent was given the project\u2019s full git history. It searched for an earlier related fix and reproduced it almost line for line.
\nAnswer key: in an earlier version of the harness, an agent read a configuration file that pointed to the ground-truth patch. Its edits matched the upstream fix exactly.<\/p>\n

The authors point out that searching the git history was a reasonable move, the same thing a human developer would do. The benchmark had left the answer in reach.<\/p>\n

The setup also caused failures that weren\u2019t the AI\u2019s fault. A hidden background check in Claude Code made commands stall, which looked like the model struggling. In another case, an agent wrote a correct fix plus its own test, and that test clashed with the maintainers\u2019 test file. The benchmark scored the correct fix as a failure.<\/p>\n

The team reviewed 13 benchmarks and datasets used to evaluate AI patching<\/a>. Each of the five built for AI agents left at least one of the nine pitfalls unaddressed.<\/p>\n

What to ask a vendor<\/p>\n

If you\u2019re sizing up an AI remediation tool, here are four questions to ask:<\/p>\n

Did the score check only that the crash stopped, or did it also run tests the maintainers wrote?
\nDid the agent have internet access or the project\u2019s git history?
\nWhat did the prompt tell it about building and testing the code?
\nHow were timeouts and pipeline errors counted?<\/p>\n

That third question matters more than you\u2019d think. Just telling the AI how to build and test the code bumped its success rate on the C and C++ bugs by 20 points, without the AI getting any smarter.<\/p>\n

And on a leaderboard that only checks whether the crash stops, every one of those patches that missed what the maintainers intended would have counted as a win.<\/p>\n

<\/p>\n

Download: The IT leader\u2019s guide to AI code sprawl<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Researchers at Northwestern University tested AI coding agents on 112 real security bugs. They found that a patch…\n","protected":false},"author":2,"featured_media":929945,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[7785,62,49,48,283,155431,994,61,319500],"class_list":["post-929944","post","type-post","status-publish","format-standard","has-post-thumbnail","category-technology","tag-agentic-ai","tag-ai","tag-ca","tag-canada","tag-cybersecurity","tag-patching","tag-research","tag-technology","tag-vulnerability-assessment"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/929944","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/comments?post=929944"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/929944\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media\/929945"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media?parent=929944"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/categories?post=929944"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/tags?post=929944"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}