It\u2019s a gradual process that happens when people in an organization take shortcuts, or don\u2019t follow the established procedures or standards, and sometimes it occurs over years. As long as nothing catastrophic happens, this deviant behavior becomes the norm.<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/au\/wp-content\/uploads\/2026\/06\/5258659.webp\")
<\/p>\n
\n Eric Brandwine, distinguished engineer and VP at Amazon Security<\/p>\n
\u201cIt\u2019s a thing all humans fall prey to, and one of the most heartbreaking stories I read in this area was about emergency departments and emergency rooms,\u201d Brandwine said during a phone interview with The Register. \u201cYou\u2019ve got all these machines, and they\u2019re all beeping. Your first day on the job, you jump every single time one of the alarms beeps \u2013 but the patient is fine. It’s a spurious alarm. You go back to your station, you sit down, and over time, after enough of these false alarms, enough of these repeated beeps with no actual consequence, your discipline slips, and you stop responding. And eventually some tragic outcome occurs.\u201d<\/p>\n
This, he admits, is a very high-stakes example. And yet it\u2019s a documented occurrence among healthcare workers<\/a>, firefighters<\/a>, and even Army pilots<\/a>.<\/p>\n \u201cLiterally, someone\u2019s life is on the line, and people still struggle to maintain discipline,\u201d Brandwine said. \u201cThat\u2019s the human condition.\u201d<\/p>\n Here\u2019s how this all applies to agentic AI governance and security. Humans build LLMs and AI systems, and having a \u201chuman-in-the-loop\u201d ensures that a person reviews the AI\u2019s output and approves (or not) any actions before the AI performs them.<\/p>\n \u201cIf you put a human inside of this tight loop, and ask them to make approval decisions for agentic tools repeatedly, time after time, they’ll do a good job,\u201d Brandwine said. \u201cAnd then they’ll do an okay job. And pretty quickly they’ll be doing a poor job.\u201d<\/p>\n This is why at Amazon, \u201cwe\u2019re not huge fans of human-in-the-loop,\u201d he added. \u201cIt’s something that you should use judiciously, where you absolutely need it. But it\u2019s not something that you can do at high velocity. You will not get the results that you want to get.\u201d<\/p>\n Big tech pulls the human-in-the-loop<\/p>\n Amazon isn\u2019t the first or only tech giant to start talking differently about the role humans should play in agentic governance.\u00a0<\/p>\n “It is very clear that we have moved from a human-led defense strategy, to a human-in-the-loop defense strategy, to an AI-led defense strategy that’s overseen by humans,” Google Cloud chief operating officer Francis deSouza told reporters<\/a> during a press conference ahead of Google’s annual Cloud Next shindig in April. “Our model for the future is an agentic fleet that does a lot of the routine cyber security work at a machine pace and then is overseen by humans.”<\/p>\n Microsoft CEO Satya Nadella, in an X missive earlier this week, argued for \u201cloop learning<\/a>,\u201d instead of having a human check an AI\u2019s output at every step.\u00a0<\/p>\n \u201cCompanies need to turn their workflows, domain knowledge, and accumulated judgment into AI systems that improve with each use,\u201d Nadella wrote. \u201cPrivate evals should capture whether a model is actually improving against outcomes that matter to the business (not just external benchmarks!). Private reinforcement learning environments should let models grow stronger on real traces from inside the organization.\u201d<\/p>\n