{"id":910901,"date":"2026-09-30T10:36:09","date_gmt":"2026-09-30T10:36:09","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/910901\/"},"modified":"2026-09-30T10:36:09","modified_gmt":"2026-09-30T10:36:09","slug":"agents-refactor-300k-lines-in-three-weeks-and-practitioners-ask-what-it-proves","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/910901\/","title":{"rendered":"Agents Refactor 300K Lines in Three Weeks, and Practitioners Ask What It Proves"},"content":{"rendered":"

CodeScene has published a case study<\/a> in which coding agents refactored a 300,000-line C codebase over three weeks, at a token cost of roughly $4,000. The work produced 2,903 commits across 726 files, modified 252,055 lines, and moved the codebase from a Code Health score of 5.6 to 10.0. The codebase is Street Fighter III: 3rd Strike, taken from an open-source decompilation.<\/p>\n

Adam Tornhill, CodeScene’s founder and the author of Your Code as a Crime Scene, wrote that this was the first time he had seen what he called superhuman AI performance at scale, after three decades working on large systems.<\/p>\n

Two mechanisms carried the work. The first was a quality signal: the CodeHealth MCP Server<\/a>, which gave agents a deterministic score to optimize and to judge whether a transformation had helped. The second was correctness: a replay-trace harness that compared the rollback state hash frame by frame, so behavior could be checked after every change.<\/p>\n

The more novel result is what the agents built along the way. Rather than applying a fixed catalogue, they accumulated a refactoring playbook, ending with 22 recipes and 82 supporting notes. Familiar transformations appear, including Extract Function and Guard Clauses, but so do recipes specific to this codebase. Shared Index Range captures repeated loops differing only in start and end ranges. Action Parameter handles duplicated control structures differing mainly in which function they invoke. Uniform Step Table converts heterogeneous calls into table-driven dispatch. Failed attempts were recorded too, including transformations that made Code Health worse.<\/p>\n

Model choice mattered. The team settled on Claude Opus for the bulk of the work, reporting that Claude Code with Opus was significantly better than Codex with Sol at capturing and documenting the emerging patterns. Files often plateaued when smaller models ran the task, appearing to reach a local optimum they could not move past.<\/p>\n

\"AI-commits-example_Streetfighter_codebase_casestudy\"<\/p>\n

Non-merge commits per day by model across the three-week project (Source: CodeScene<\/a>)<\/p>\n

Reaction from practitioners on LinkedIn<\/a> has been sharply divided, and the split runs along what the result proves rather than whether it happened.<\/p>\n

Paolo Perrone put the case for taking it seriously:<\/p>\n

<\/p>\n

most refactor claims i’ve read rest on a green test suite, which only tells you the tests survived. replaying traces against a fighting game sets a much higher bar.<\/p>\n

<\/p>\n

Mats Iremark, CTO at Omda Response, reported doing something comparable, writing that the CodeScene MCP combined with current agents is almost like cheating.<\/p>\n

The skeptics concentrated on scope. Konrad Otr\u0119bski, a tech lead and consultant, asked whether the work was merged, whether it arrived as one enormous merge or many, and whether this was an experiment on open-source code rather than production code earning money. Daniel Webb, CTO at NeoSee and one of the two engineers who did the work, replied that it was merged to main on a fork, through 54 pull requests.<\/p>\n

Otr\u0119bski’s follow-up set a higher bar:<\/p>\n

<\/p>\n

I think the real true test of AI capability here would be to offer this courtesy of refactoring to some famous open source project, say Grafana. The definition of done is ofc merging it to master.<\/p>\n

<\/p>\n

Tracy Bannon, a software architect and researcher, objected to the framing rather than the method, noting that describing the outcome as perfect is pretty bold.<\/p>\n

Denis Baltor questioned the discovered recipes themselves, arguing that DRY concerns duplication of knowledge and intent rather than identical lines of code, and that recipes defined by loops differing only in ranges, or control structures differing only in the function invoked, may be collapsing the two.<\/p>\n

Asko N\u00f5mm raised the methodological question. Since Claude Code and Codex are tuned to their own models, he argued it is unclear how much of the result measures the model and how much measures the harness, and that cost would vary the same way. He added that architecture remains unmeasured, so code can look healthy while fundamental problems surface later.<\/p>\n

Several unanswered questions came from the authors themselves. Marc Bouvier asked whether non-functional behavior had improved, given that framerate, memory use and input latency matter in a game; Webb said a performance specialist was being brought in. Asked whether the harness caught subtle frame timing regressions, he said there might be no definitive answer, because it ran as a pre-commit hook and some failures were fixed without being observed. On diff sizes, he was equally direct: if you are not familiar with the code and no longer review every line, how big can a diff be? He made no assertion, only questions.<\/p>\n

If these reactions are representative, the disagreement is not about the numbers but about the conditions that produced them. The replay-trace harness worked because a decompiled game offers deterministic frame-by-frame replay. Most legacy systems have no equivalent oracle, which is the same reason refactoring them is risky in the first place. Tornhill writes that automated tests and equivalence checks are absolutely essential safeguards, which places the burden on exactly the thing unhealthy codebases tend to lack.<\/p>\n

The team’s own selection process illustrates it. Webb said they considered a Gov.UK marine licensing codebase he had worked on, but it was too healthy to be useful for the research that follows, and chose the game partly because they play it and are therefore its users.<\/p>\n

That research is the next step. The uplift produced two functionally equivalent versions of the same system, one at Code Health 5.6 and one at 10.0, for a study with Lund University in which students will implement features in both using frontier models and compare cost and quality.<\/p>\n

Two headline figures belong to that future work rather than this case study. CodeScene projects a roughly 70% reduction in AI-induced defects and roughly 45% less token waste from the uplift, both extrapolated from its earlier research rather than measured here. The $4,000 and the three weeks were measured.<\/p>\n","protected":false},"excerpt":{"rendered":"CodeScene has published a case study in which coding agents refactored a 300,000-line C codebase over three weeks,…\n","protected":false},"author":2,"featured_media":910902,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[31],"tags":[478693,256,28302,19937,449,458,459,64,63,478696,460,16553,134,19938,478695,31974,478694],"class_list":["post-910901","post","type-post","status-publish","format-standard","has-post-thumbnail","category-arts-and-design","tag-agentic-refactoring-case-study","tag-ai","tag-ai-development","tag-architecture-design","tag-arts","tag-arts-and-design","tag-artsanddesign","tag-au","tag-australia","tag-code-quality","tag-design","tag-development","tag-entertainment","tag-ml-data-engineering","tag-refactoring","tag-software-engineering","tag-technical-debt"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/910901","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=910901"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/910901\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/910902"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=910901"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=910901"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=910901"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}