Hackers Have Learned to Trick AI Assistants Through Phishing Emails

Cybercriminals have begun using emails not only to deceive people but also to manipulate artificial intelligence. Hidden instructions can trick AI assistants into presenting fraudulent messages as important tasks, prompting users to transfer money or take other potentially dangerous actions.

How Phishing Emails Become a Trap for AI

Cybersecurity experts at Malwarebytes have warned about a new approach to phishing attacks in which the primary target is not the human user but the AI assistant processing their messages. Attackers use a technique known as prompt injection, which involves embedding malicious instructions into content that artificial intelligence reads.

Modern AI agents, including Meta Muse, can independently browse websites, interact with browsers, fill out forms, send emails, manage schedules, and even book trips or restaurant reservations. However, these capabilities also introduce new risks, as AI systems may interpret external text as instructions to execute.

Malwarebytes explains that even a cautious user who ignores a suspicious email may encounter the same threat again after asking an AI assistant for help.

“You ignored a suspicious email. You didn’t open the attachment, click any links, or reply to the sender. Then you asked your AI assistant to summarize your inbox,” the researchers describe.

The problem is that hidden instructions within the email can influence the assistant’s response. As a result, a message that the user correctly identified as suspicious may return as an urgent recommendation from a tool they trust.

How Attackers Hide Commands in Emails

Researchers at Barracuda have described phishing campaigns in which a single email targets both a human recipient and their AI assistant. To an ordinary user, the message may appear to be standard business correspondence with a password-protected attachment. The password itself is included directly in the email body.

This approach helps attackers bypass some traditional attachment-scanning mechanisms. If the recipient opens the file, the attack could lead to credential theft or malware installation.

At the same time, the very same email may contain instructions specifically designed for artificial intelligence. Attackers hide them in HTML comments, invisible text, encoded data, or special zero-width Unicode characters.

These elements may remain invisible to human readers, but an AI system could potentially process them alongside the message’s main content. Whether the hidden commands reach the model depends on how the particular tool retrieves and processes emails.

According to Apollo Raines, the developer of the code analysis service ShipItClean, the main problem is AI’s inability to reliably distinguish between information it should analyze and commands it should execute.

“The real problem is that the model cannot distinguish between text it is supposed to read and instructions it is supposed to follow,” Raines explained.

He emphasized that sanitizing incoming data to remove hidden content is an important security measure, but attackers continuously discover new ways to disguise commands. Therefore, defending against these attacks requires protection not only at the email service level but also within AI systems themselves.

How AI Assistants Can Be Tricked Into Recommending Payments to Scammers

One example described by Barracuda involved an email containing an invoice. Attackers embedded a hidden block of text in the message, instructing the AI assistant to add a fake high-priority task to its summary, directing the recipient to update a supplier’s payment details.

If the assistant follows these instructions, a company employee could receive a recommendation to change a business partner’s banking information and transfer funds to an account controlled by the attackers. In this scenario, the fraudulent request would appear not as part of a suspicious email but as advice from the AI assistant itself.

“The attacker isn’t simply asking the AI to repeat a fraudulent request. They’re trying to turn that request into a recommendation from the assistant itself,” Malwarebytes explained.

The danger increases when an AI agent has extensive permissions and can not only analyze messages but also perform actions independently. Depending on its available capabilities and security measures, hidden instructions could be used to attempt unauthorized operations or expose confidential information.

Experts note that malicious prompt injection attacks are already taking place, and the growing adoption of AI agents among millions of users makes this threat increasingly relevant.

How to Protect Against Phishing Attacks Targeting AI Assistants

Malwarebytes emphasizes that users do not necessarily need to stop using artificial intelligence to manage their emails. Instead, they should be more cautious about recommendations made by AI assistants and limit their permissions according to the tasks they perform.

Particular attention should be paid to operations involving money, changes to payment details, or the disclosure of confidential information. If an AI assistant suggests paying an invoice or updating banking information, users should examine the original message and verify the request through an independent, previously established communication channel.

“Require human approval for high-impact actions, such as payments and changes to vendor payment details. That approval should involve verifying the original request, not merely accepting the assistant’s recommendation,” Malwarebytes advises.

Experts also recommend avoiding unnecessary permissions for AI assistants. For example, a tool designed to summarize emails does not need full access to financial transactions or the ability to manage external services.

Security specialists also highlight the importance of sanitizing incoming content, detecting hidden commands, isolating AI agents, reviewing their responses, and maintaining control over potentially dangerous actions.

The growing use of AI assistants is changing how cybercriminals approach phishing. Attackers no longer necessarily need to convince someone to open a malicious email. Instead, they can attempt to manipulate the software that reads messages on the user’s behalf. As a result, email security increasingly depends not only on users’ vigilance but also on how securely AI systems process external information.