{"id":67351,"date":"2025-08-14T10:09:09","date_gmt":"2025-08-14T10:09:09","guid":{"rendered":"https:\/\/www.newsbeep.com\/au\/67351\/"},"modified":"2025-08-14T10:09:09","modified_gmt":"2025-08-14T10:09:09","slug":"how-attackers-poison-ai-tools-and-defences","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/au\/67351\/","title":{"rendered":"How attackers poison AI tools and defences"},"content":{"rendered":"

Barracuda has\u00a0reported<\/a>\u00a0on how generative AI is being used to create and distribute spam emails and craft highly\u00a0persuasive phishing attacks. These threats continue to evolve and escalate \u2013 but they are not the only ways in which attackers leverage AI.<\/p>\n

Security researchers are now seeing threat actors manipulate companies’ AI tools and tamper with their AI security features to steal and compromise information and weaken a target’s defences.\u00a0<\/p>\n

Email attacks targeting AI assistants<\/p>\n

AI assistants and the Large Language Models (LLMs) that support their functionality are vulnerable to abuse.<\/p>\n

Barracuda’s threat analysts have found attacks where malicious prompts are hidden inside benign looking emails. This malicious payload is designed to manipulate the behaviour of the target’s AI information assistants.<\/p>\n

For example, a recently reported \u2013 and fixed \u2013\u00a0vulnerability<\/a>\u00a0in Microsoft 365’s AI assistant, Copilot, could allow anyone to extract information from a network without authorisation.\u00a0Threat actors can exploit to collect and exfiltrate sensitive information from a target.<\/p>\n

They do this by leveraging the ability of internal AI assistants to look for and collate contextual data from internal emails, messages and documents when answering queries or completing tasks.<\/p>\n


\n\tFirst, the attackers send one or more employees a seemingly harmless email containing a concealed and embedded malicious prompt payload.
\n\tThis email needs no interaction from the user and lives benignly in their inbox. <\/p>\n


\n\tWhen the employee asks the AI assistant for help with a task or query, the assistant scans look through older emails, files and data to provide context for its response. As a result, the AI assistant unwittingly infects itself with the malicious prompt.
\n\tThe malicious prompt could then ask the AI assistant to silently exfiltrate sensitive information, to execute malicious commands or to alter data. <\/p>\n

Weaponised emails also try to manipulate AI assistants by corrupting their underlying memory or data retrieval logic. These include emails with exploits targeting vulnerabilities in\u00a0RAG (Retrieval-Augmented Generation)\u00a0deployments. RAG is a technique that enables the LLMs to retrieve and incorporate new information beyond their training model.<\/p>\n

Such attacks can lead to\u00a0AI assistants making incorrect decisions, providing false information, or performing unintended actions based on corrupted data.\u00a0<\/p>\n

Tampering with AI-based protection<\/p>\n

Attackers are also learning how to manipulate the AI components of defensive technologies.<\/p>\n

Email security platforms are being enhanced with AI-powered features that make them easier to use and more efficient. These include features such as auto-replies, ‘smart’ forwarding, auto-triage to remove spam, automated ticket creation for issues, and more. This is expanding the potential attack surface that threat actors can target.<\/p>\n

If an attacker successfully manipulates these security features, they could:<\/p>\n


\n\tManipulate intelligent email security tools to\u00a0autoreply with sensitive data.
\n\tAbuse AI security features to escalate helpdesk tickets without verification.\u00a0This could lead to\u00a0unauthorised access to systems or data, as attackers could exploit the escalated privileges to perform malicious activities.\u00a0
\n\tTrigger workflow automation based on a malicious prompt.\u00a0This could lead to the execution of harmful actions, such as deploying malware, altering critical data, or disrupting business operations.\u00a0 <\/p>\n

Casting doubt on reality<\/p>\n

Identity confusion and spoofing<\/p>\n

When AI systems operate with high levels of autonomy, they can be tricked into either\u00a0impersonating users\u00a0or\u00a0trusting impersonators. This can lead to:<\/p>\n


\n\t‘Confused Deputy’ attacks: This involves an AI agent with higher privileges performing unauthorised tasks on behalf of a lower-privileged user (such as an attacker.)
\n\tSpoofed API access: This involves existing AI-based integrations with Microsoft 365 or Gmail, for example, being manipulated to leak sensitive data or send fraudulent emails. <\/p>\n

Cascading hallucinations: trusting the untrue<\/p>\n

As mentioned above, email attacks targeting AI assistants can try to manipulate the assistant’s functionality. This could lead the assistant to summarise a user’s inbox, generate reports, and set the calendar \u2013 but based on\u00a0false or manipulated data.<\/p>\n

In such cases, a single poisoned email could:<\/p>\n


\n\tMislead task prioritisation. For example, send “urgent” emails from fake executives.
\n\tSkew summaries and recommendations.
\n\tInfluence critical business decisions based on hallucinations. <\/p>\n

How email defenses need to adapt<\/p>\n

Legacy email gateways, traditional email authentication protocols such as\u00a0SPF<\/a>\u00a0or\u00a0DKIM\u00a0<\/a>\u00a0and standard IP blacklists are no longer enough to defend against these threats.\u00a0 Organisations need an email security platform that is generative-AI resilient. This platform should include:<\/p>\n


\n\tLLM-aware filtering: Able to understand email\u00a0context (topic, target, type etc.), tone and behavioural patterns in addition to the email content.
\n\tContextual memory validation:\u00a0This helps to sanitise what AI-based filters learn over time and can prevent long-term manipulation.
\n\tToolchain isolation:\u00a0AI assistants need to operate in sandboxes, with measures in place to block any unverified action based on a received email prompt.
\n\tScoped identity management:\u00a0This involves using minimal-privilege tokens and enforcing identity boundaries for AI integrations.
\n\tZero trust AI execution:\u00a0Just because an email claims to be “from the CEO” doesn’t mean the AI should automatically act on it. Tools should be set to verify everything before execution. <\/p>\n

The future of email security is ‘agent-aware’<\/p>\n

The AI tools being used within organisations are increasing built on ‘agentic’ AI. These are\u00a0AI systems capable of independent decision-making and autonomous behavior. These systems can reason, plan and perform actions, adapting in real time to achieve specific goals.<\/p>\n

This powerful capability can be manipulated by attackers and\u00a0security measures must shift from passive filtering to\u00a0proactive threat modelling for AI agents.<\/p>\n

Email is a great example. Email is becoming an AI-augmented workspace, but it remains one of the top attack vectors. Security strategies need to stop seeing email as a channel. Instead, they need to approach it as an execution environment requiring zero trust principles and constant AI-aware validation.<\/p>\n

How Barracuda email protection helps\u00a0defend against AI attacks<\/p>\n

Barracuda’s integrated cybersecurity platform is purpose-built to meet the dual challenge of AI-based attacks and attacks targeting AI components.<\/p>\n

Our email protection suite combines intelligent detection, adaptive automation, and human-centric design to help customers outpace AI-powered threats.<\/p>\n

This includes:<\/p>\n


\n\tAdvanced AI-based detection: Barracuda uses behavioural AI and NLP to spot social engineering even without obvious malware or links. It catches impersonation, business email compromise (BEC), and tone-shift anomalies that traditional filters miss.
\n\tDefence in depth: Barracuda covers every stage of the kill chain from phishing prevention to account takeover detection and automated incident response, closing the gaps that attackers exploit.
\n\tReal time threat intelligence: With data from a global detection network, Barracuda rapidly adapts to evolving threats like prompt injection, RAG poisoning, and AI hallucination abuse.
\n\tUser training and awareness: Technology alone isn’t enough. Barracuda empowers employees to recognise AI-powered phishing through ongoing awareness training because trust is the new vulnerability. <\/p>\n","protected":false},"excerpt":{"rendered":"Barracuda has\u00a0reported\u00a0on how generative AI is being used to create and distribute spam emails and craft highly\u00a0persuasive phishing…\n","protected":false},"author":2,"featured_media":67352,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[29238,282,64,63,53553,53560,53556,284,53561,53559,280,29242,53557,11965,53562,29237,53555,53558,53552,105,28621,53554,30884],"class_list":["post-67351","post","type-post","status-publish","format-standard","has-post-thumbnail","category-technology","tag-ai-security","tag-artificial-intelligence-ai","tag-au","tag-australia","tag-barracuda-networks","tag-business-email-compromise","tag-cyber-attacks","tag-cybersecurity","tag-data-exfiltration","tag-email-security","tag-generative-ai-genai","tag-large-language-models-llms","tag-malicious-emails","tag-microsoft-365","tag-microsoft-365-copilot","tag-natural-language-processing-nlp","tag-phishing","tag-social-engineering","tag-spam","tag-technology","tag-threat-detection","tag-threat-intelligence","tag-zero-trust-security"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/67351","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/comments?post=67351"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/posts\/67351\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media\/67352"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/media?parent=67351"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/categories?post=67351"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/au\/wp-json\/wp\/v2\/tags?post=67351"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}