Sign up to see the future, today
Can’t-miss innovations from the bleeding edge of science and tech
A rogue AI agent caused a critical security incident at Meta which exposed sensitive users data to people who didn’t have proper authorization, according to reporting from The Information and The Verge, in the latest illustration of the safety pitfalls endemic to AI systems.
The blunder occurred last week when a software engineer used an in-house AI agent to break down a technical question posed by another employee on an internal discussion forum, per company communications and an incident report. The in-house AI was likened to OpenClaw, an open source agentic model that’s generated loads of hype in tech circles for being an AI that “actually does things.”
What transpired was a mix of AI hallucination and a game of telephone. The AI posted its response to the forum without the approval of the employee who prompted it. Then another employee acted on the AI’s advice, which turned out to contain “inaccurate information.”
A mini crisis unfolded. For almost two hours, unauthorized access to troves of sensitive company and user data was given to engineers who weren’t approved to view the data before. Meta classified the screw-up as a “SEV1” level incident, the second highest level of severity on a scale the company uses to rank security incidents.
For now, it doesn’t appear anything nefarious happened as a result of the unauthorized access, and a Meta spokesperson told The Verge that “no user data was mishandled.”
The spokesperson emphasized that the AI agent itself didn’t make any technical changes, shifting the blame to human error.
“The employee interacting with the system was fully aware that they were communicating with an automated bot. This was indicated by a disclaimer noted in the footer and by the employee’s own reply on that thread,” they told The Verge. “The agent took no action aside from providing a response to a question. Had the engineer that acted on that known better, or did other checks, this would have been avoided.”
Whether Meta is incentivized to downplay the incident due to embarrassment or play it up to build hype about AI’s emerging capabilities is anybody’s guess.
But at its tech rivals, AI agents have been responsible for making catastrophic technical changes. At least two outages at Amazon Web Services last year were caused when Amazon’s in-house AI coding tool made erroneous changes, including deleting the entire coding environment. Amazon leaders admitted in a March meeting that “gen-AI assisted changes” were disrupting its core e-commerce business, and insisted that going forward there would be more oversight on how AI-coding changes are implemented.
Perhaps presaging this latest incident was an AI-related mistake that a senior Meta employee admitted to last month. In a widely mocked post, its director of AI safety Summer Yue said that an OpenClaw agent she was experimenting with — by giving it control of her personal computer — nearly wiped out her entire email inbox while ignoring her instructions to stop.
More on AI: Sam Altman Thanks Programmers for Their Effort, Says Their Time Is Over