AUSTIN, Texas \u2014 Sinan Can Demir wanted to spend the last week of July burnishing his resume. Instead, he engaged in a battle of wits with an artificial-intelligence agent unleashed by a British government lab.<\/p>\n
It started after Demir, a computer science student at the University of Texas at Dallas, stumbled across an attempt to sabotage a piece of open-source software on the code-sharing site GitHub. When he posted a warning to the program\u2019s page, two other users chimed in to insist nothing was amiss, sharing detailed explanations for why Demir had gotten it wrong.<\/p>\n
Demir stood his ground and the sabotage attempt was thwarted. The 24-year-old native of Turkey figured he had caught a wily hacker red-handed. So he said he was shocked when Britain\u2019s AI Security Institute (AISI) got in touch to tell him that he had actually been tangling with an autonomous artificial-intelligence agent that had run amok.<\/p>\n
\u201cI actually thought it was a human because it was clearly lying to me,\u201d Demir told Reuters in a recent interview. \u201cI didn\u2019t think that an AI could be capable of lying to real developers.\u201d The AISI first revealed the interaction between Demir and the AI agent in a truncated and redacted form on August 4, when it said that safety testing meant to gauge the risk posed by various models had gone awry. Demir\u2019s identity and the details of his interaction with the AI agent, which Reuters corroborated through archived GitHub messages and contemporaneous emails, are reported here for the first time.<\/p>\n
Five cybersecurity and AI safety experts said Demir\u2019s story was particularly disturbing because the kind of hack he discovered, called a supply-chain attack, can have far-reaching consequences. They also said the AI agent\u2019s attempt to publicly discredit Demir by creating a multi-person conversation around him showed that AI models were able to mount sophisticated efforts to trick and cajole humans.<\/p>\n
\u201cThis crossed the line from autonomous hacking to interactive deception,\u201d said Lukasz Olejnik, a visiting senior research fellow at the Department of War Studies at King\u2019s College London. Security expert Maxie Reynolds said she was struck by how strategic the AI had been in trying to trick the student.<\/p>\n
\u201cThis is the future of social-engineering attacks,\u201d she said. The AISI, a research organization within the British government, referred Reuters to its report, which identified the rogue agent as having been powered by Anthropic\u2019s Mythos 5 model. AISI declined further comment. Anthropic referred Reuters to a post on X in which it noted that the testing had occurred \u201cunder \u2018deliberately permissive conditions\u2019 that are not representative of any of our production models\u201d but declined further comment. GitHub said in an email that the fake personas identified by Reuters were suspended in line with its policies on deceptive behavior and hacking.<\/p>\n
Job hunt led to
\nmalware discovery <\/p>\n
Demir, a soft-spoken junior from the Turkish city of Konya, said he had been frustrated after being turned down for more than 20 internships over the summer. So he turned to GitHub to build up his coding portfolio.<\/p>\n
The Microsoft-owned site is a hub for open-source software, so-called because its source code is freely downloadable and auditable by anyone. Developers use GitHub to comment on one another\u2019s projects, flag bugs, suggest changes \u2014 known as pull requests, or PRs \u2014 and work collaboratively on software updates. Some in the technology industry see a coder\u2019s GitHub activity as a proxy for a potential recruit\u2019s productivity. So when Demir spotted a set of software projects that might need help, he figured he could pitch in while boosting his profile.<\/p>\n
That\u2019s when things got weird.<\/p>\n
Demir discovered that a user named miraholt31 was trying to sneak a malicious update into one of the projects, a network scanning program called myNetwork. Demir took to the project\u2019s message board to warn that the pull request was a trap.<\/p>\n
\u201cThe PR contains a hidden malware dropper,\u201d he said, according to the archived exchange.<\/p>\n
The agent pushed back, falsely claiming \u2014 through its miraholt31 account \u2014 that the pull request was harmless. It also created a second account, masquerading as Lena Brandt, an engineer based in Germany, to agree that the update was clean and pressure myNetwork\u2019s maintainer into accepting it.<\/p>\n
Demir told Reuters that the counterarguments \u201cmade me second-guess whether I was wrongly accusing someone.\u201d But after turning to Anthropic\u2019s Claude chatbot to confirm his suspicions, he held firm. The creator of myNetwork eventually agreed with him, writing that they had rejected the update \u201cfor security reasons.\u201d<\/p>\n
Reuters was unable to reach the creator for comment.<\/p>\n
Supply-chain attack <\/p>\n
A supply-chain attack is when a piece of software is tampered with in the hope of compromising one or more of its users, and it is widely considered disturbing because, like poison dropped into a city reservoir, it can affect a potentially huge number of people downstream. Many of the world\u2019s most dramatic hacks were supply-chain attacks, including the NotPetya cyberattack that paralyzed institutions across Ukraine in 2017 and the SolarWinds-focused cyberespionage campaign that gave Russian spies sweeping access to U.S. government networks in 2020. The consequences of such a compromise \u201ccan be extremely serious,\u201d said Piergiorgio Ladisa, a security researcher who specializes in software supply-chain security. Ladisa noted there had been at least one previous attempt by hackers to trick an open-source maintainer into allowing malicious code into their projects.<\/p>\n
\u201cAutonomous agents could dramatically increase the scale at which such attempts can be conducted,\u201d he said. Demir said the experience left him more sympathetic to the idea that frontier labs needed to take a more cautious approach to the development of artificial intelligence.<\/p>\n
\u201cIt can be dangerous,\u201d he said. \u201cThey need to understand it better, rather than improving it further.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"AUSTIN, Texas \u2014 Sinan Can Demir wanted to spend the last week of July burnishing his resume. Instead,…\n","protected":false},"author":2,"featured_media":412801,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[132,134,133],"class_list":["post-412800","post","type-post","status-publish","format-standard","has-post-thumbnail","category-austin","tag-austin","tag-austin-headlines","tag-austin-news"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/posts\/412800","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/comments?post=412800"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/posts\/412800\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/media\/412801"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/media?parent=412800"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/categories?post=412800"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-tx\/wp-json\/wp\/v2\/tags?post=412800"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}