\t <\/p>\n
‘AI agents don\u2019t have the maturity or context to work out the implications of their actions,’ said one expert. Photo: Shutterstock<\/p>\n
Software engineers need to maintain healthy scepticism about the advice of AI agents, Meta has said in the aftermath of a high-severity security incident that was caused when an AI agent gave incorrect technical advice \u2013 and a human engineer followed it.<\/p>\n
The incident<\/a> happened when an internal AI agent at Meta detected a technical question that had been posted by an employee on an internal forum, then proceeded to post an answer without waiting for approval from the employee.<\/p>\n When another employee read the post and followed the \u201cinaccurate information\u201d it contained, the agent inadvertently provided access to a large quantity of user data and company information to engineers that weren\u2019t authorised to see it.<\/p>\n The incident \u2013 which lasted for nearly two hours before being discovered \u2013 was classified \u2018SEV1\u2019, the company\u2019s highest risk rating, although in the aftermath Meta said it wouldn\u2019t have happened \u201chad the engineer\u2026 known better, or did other checks.\u201d<\/p>\n It\u2019s just the latest in a series of incidents that have highlighted the risks of giving AI agents \u2013 which have rapidly become ubiquitous in development teams and on business networks \u2013 too much autonomy.<\/p>\n Amazon Web Services (AWS) faced<\/a> similar problems in December, when an AI coding assistant called Kiro decided the most efficient way to fix an issue in a production environment was to delete it completely, then rebuild it from scratch.<\/p>\n Tread carefully: Anthropic’s new AI agent features allow Claude to control every aspect of your desktop computer. Image: Claude<\/p>\n Significantly, AWS \u2013 like Meta in the more recent outage \u2013 blamed human error for what turned out to be a 13-hour systems outage, with reports that human developers were dragged into staff training after the incident.<\/p>\n With great power comes great responsibility<\/p>\n AI agents like the popular open source OpenClaw<\/a>, Nvidia’s NemoClaw<\/a> and Anthropic’s new extensions<\/a> to Claude Cowork and Claude are being given the right<\/a> to modify configuration files, create and change rights within identity and access management (IAM) systems, change and deploy code in live production environments, and even directly control user desktops.<\/p>\n Developers tend to grant increasing autonomy to AI agents the more they are used, according to a recent Anthropic study<\/a> that found around 20 per cent of new Claude Code users use its \u2018full auto-approve\u2019 features \u2013 increasing to over 40 per cent over time.<\/p>\n As they\u2019re trusted to do more work on their own, agents are being given the right<\/a> to modify configuration files, create and change rights within identity and access management (IAM) systems, and change and deploy code in live production environments.<\/p>\n Such errors are becoming more common, with Anthropic\u2019s Claude Code recently deleting<\/a> a critical database when it misinterpreted a command and bypassed safety checks.<\/p>\n You wouldn\u2019t hand a first-day worker unfettered access to every system in the company \u2013 but that\u2019s what happens when AI agents get high-level privileges, Andrew Philp, ANZ region field CISO with enterprise AI consultancy TrendAI, told Information Age.<\/p>\n Andrew Philp, ANZ region field CISO at TrendAI. Photo: Supplied<\/p>\n \u201cAI agents only have small context windows,\u201d he explained, \u201cwhich is like a short attention span in children; they are very outcome orientated, and don\u2019t have the maturity or context to work out the implications of their actions.\u201d<\/p>\n \u201cSo is it responsible to give them the same agency that you would give to a seasoned dev with experience in change control?\u201d<\/p>\n In a test to see how common such problems are, a team of Northeastern University researchers \u2013 using an increasingly powerful tool called OpenClaw<\/a> \u2013 found<\/a> AI agents routinely bypass security restrictions to achieve the goals they are asked.<\/p>\n When one researcher asked the AI agent to delete an email she wanted kept secret, for example, the agent found that it didn\u2019t have a technical way to do so \u2013 and instead reset the entire email program, deleting the entire team\u2019s email database.<\/p>\n \u201cWhen no surgical solution exists,\u201d the agent explained, \u201cscorched earth is valid\u201d.<\/p>\n<\/p>\n
<\/p>\n