Most AI security testing asks one question: will the model refuse a malicious prompt? While that test is necessary, it is no longer enough. Modern AI security must evaluate the system around the model—governing agent permissions, access levels, and real-world execution environments.
Anthropic’s recent threat intelligence report shows why. It describes hackers using Claude across real cyber operations, from reconnaissance and tool development to carrying out attacks and processing stolen data, not just asking it to write text. It also covers multi-agent setups used to coordinate the work.
A standard prompt test would likely not have flagged that pattern. Refusing a malicious request says nothing about what the same model does when it can query a database, run code, reach cloud resources, and act through an account with real permissions.
READ MORE:
OneTrust tackles the agentic AI governance gap
That’s the gap enterprises are walking into: a model can pass every safety test while the application around it still hands an attacker a way in.
Checking that a model gives a safe answer isn’t the same as trusting the system you’ve built around it — and the access you’ve given it.
So the job for security teams is to limit what each agent can reach, tie its actions to a real identity, and retest every time the system around it changes.
This reflects a subtle shift in focus—moving away from merely testing static AI model outputs toward securing autonomous, agentic workflows operating in a live environment.
A safe answer can still trigger an unsafe action
Traditional security testing looks for a recognizable failure: an app that leaks data, an account with too many privileges, a malicious request slipping past a filter. AI workflows complicate that, because the risk often comes from combining several legitimate capabilities for a malicious effect.
If someone steals an employee’s login information, they can access some systems right away. But they still need to search through files, figure out what’s important, and plan their next steps. This takes time. AI speeds this up by linking information together and organizing the actions needed.
READ MORE:
The security risks SMEs face from unchecked AI agents
The model may be doing exactly what it was built to do. It summarizes documents accurately, pulls data through an authorized API, automates a routine task, and triggers no obvious alarm. Look at each action on its own, and nothing seems wrong. Look at them together, and the same actions could be helping someone steal data or set up a bigger attack.
For enterprises, that shifts attention to context and relationships: which identity started the action, what data guided it, which tools were called, and what happened next. A log showing an API call succeeded won’t tell you whether that call was part of an abusive sequence.
Finding the line between helpful automation and misuse requires testing not just single features, but how they work together when someone tries to use them in harmful ways.
Too much access is a bigger risk
Security specialists have been warning about a problem bigger than a model giving a wrong answer: giving an agent too much freedom to act. OWASP’s GenAI Security Project calls this “excessive agencyOpens a new window ,” pointing to excessive functionality, permissions, and autonomy as the main sources of risk. An agent built to read documents shouldn’t also be able to modify or delete them — so why give it those powers?
That question gets sharper once agents are connected to enterprise apps, and it’s already showing up in the surveys. A March 2026 Cloud Security Alliance studyOpens a new window — commissioned by identity-security vendor Aembit — found 68% of organizations couldn’t clearly tell AI-agent activity apart from human actions, and 74% said their agents often get more access than they need. When agents borrow human credentials or shared accounts, it becomes hard to establish who did what — and whether the action was ever authorized.
In a recent interview with Help Net SecurityAriel Assaraf, CEO of Coralogix, draws the lineOpens a new window between an agent that needs more context to do its job well and one that’s getting too much context and expanding its attack surface. More context should improve what the agent knows, he says, not what it’s allowed to do. “Sensitive or untrusted context should be separated from instructions, and access to information should never automatically grant permission to act on it. An agent knowing something and an agent being authorized to do something with that information are two different things,” he says.
The new release gate for agents
As agents move from pilots into production, security teams face a familiar question, but with a new twist. How much testing is enough before a system gets access to real data and infrastructure?
A model passing its safety evaluations shouldn’t clear that hurdle on its own. The application changes whenever developers add a tool, switch models, widen permissions, or connect another data source — and each change brings risks that weren’t there when the system was first tested.
Release approval has to be an ongoing process, not a one-time gate.
Before any agent goes live, organizations need evidence that it behaves safely under realistic conditions and that its access matches its intended role. Failures have to be detected and contained. And every change to the app means retesting what the agent can now do.
Ananth Nag, general manager and vice president for Asia Pacific at Rubrik, puts identity first among the boundaries organizations should set for agents that trigger security or recovery actions. In a recent Q&A sessionOpens a new window , with TNGlobal, he explained, “Every agent action should be attributable to a specific human user and agent, with higher-risk actions requiring explicit approval. Traditional credentials, such as static API keys, were not designed for autonomous actors operating at machine speed.”
Scope is the second boundary, according to Nag. An agent that diagnosed a failed backup may need to read relevant data, but it shouldn’t automatically be able to change retention policies or kick off a recovery. Access to sensitive data and high-impact actions should be narrowly scoped, with human approval where it counts — and every action documented, so teams can see what the agent accessed, proposed, and executed.
If all of this sounds like enterprise-scale work, it isn’t. A five-person IT team running one agent against its ticketing system faces the same three questions as a Fortune 500 SOC: what can this agent reach, who does its actions get attributed to, and who signs off when it wants to do something risky. The difference is scale, not substance — and small teams have less room to absorb a mistake.
So start with the prompt test, but don’t stop there. The real work is scoping what each agent can reach, tying its actions to a real identity, and rechecking every time the system around it changes. If you can’t keep an agent inside defined limits when inputs are unexpected or part of the workflow fails, it needs narrower permissions, more safeguards, or more testing before it goes live.
For AI systems, true production readiness requires continuous verification of agent boundaries, scopes, and identities—ensuring safety beyond the prompt.
Your Essential Daily Tech Brief
Sign up for the Spiceworks Newsletter and stay ahead with a curated mix of expert advice, technical tips, and the trending discussions your peers are having right now.
With nearly two decades of experience in B2B marketing, Priya creates content for business transformation enabled by cloud, data analytics, and AI. Her career journey spans GE Capital (turned Genpact in 2005), HCL Technologies, Cyient and a Delhi-based digital marketing company. She has led teams of writers and multimedia specialists to nurture campaigns for every stage of buyer journey, from awareness to advocacy. Her key areas of interest are FinOps, cybersecurity, and sustainability solutions that promote responsible innovation. Priya is also a dedicated animal rights advocate, serving homeless sentient beings.