Incident Response When the Attacker Also Uses AI

Phishing messages once gave themselves away through clumsy grammar and generic greetings. Those signs are weaker now, and response plans that depended on them need rewriting. The message that reaches a finance inbox may read exactly like a colleague wrote it, and the first decision in an incident has to be made without that signal.

How attacker use of AI changes an incident

Attackers can now produce plausible messages in many languages, in a tone that matches the sender, and in many variants. That affects an incident in three ways. Volume increases, because drafting a campaign takes less effort. Variation increases, because each message can differ enough to slip past a rule written for one template. And the quality of first contact improves, so a message is less likely to be dismissed on sight.

What does not change is the goal. Attackers still want credentials, money, data or access, and they still use familiar routes: a compromised identity, a mailbox rule that hides activity, an unpatched system or a trusted third party. Containment still means cutting off access and preserving evidence. The scale of the change is hard to measure, and it is worth being careful about claims that it has transformed every attack. What has changed is how easily a convincing message can be produced.

Triage on behavior, not on the message

If message quality no longer separates benign from malicious, triage has to rely more on behavior. Look for unusual sign-in patterns, new forwarding or inbox rules, requests for payment or access that arrive through an unusual channel, and data access that falls outside a user's normal work. Each of these can be checked against logs without judging how the message reads.

Automated enrichment helps group related messages, sender infrastructure and affected accounts quickly. The scope decision should still belong to a person, because automation tends to find what it was built to find. Add a verification step to any request that asks for a change, using a channel the requester did not supply.

Grouping matters as much as detection. Varied wording still shares fields that are harder to vary: a sending domain, a payment destination, a request type, a reply address. Search mailboxes on those fields rather than on the text, and check who clicked, replied or forwarded before anyone is told the campaign is over. Early estimates of scope can be too small, because the recipients who noticed first are the ones who reported.

Write the decision points in advance

Response slows down when people have to work out during an incident who can do what. Write the decision points in advance: who can disable an account, when a host is isolated, when legal counsel is informed, when customers are told and who approves that message.

AI features need their own decision points. Decide before any incident whether a feature can be switched off, who holds that authority and how the feature is restored safely. A kill switch that nobody has tested is not a control. The same applies to tool access. If an agent is suspected of acting on injected content, the response team should be able to revoke its credentials without waiting for the feature team to diagnose the cause.

Restoring a feature needs its own step. Before switching it back on, confirm what triggered the problem, check that the affected tool permissions are narrower than before, and read a sample of the actions taken during the window. A feature restored on the same settings that failed tends to fail again, and the second failure is usually harder to explain than the first.

Communicate with staff clearly

Staff need short, specific guidance. Tell them what to check, such as whether a request arrived through the usual channel and whether it asks for something unusual, rather than asking them to judge whether a message looks real. Give one route for reporting suspicious contact, and confirm to the person who reported that the report was received. A single warning fades quickly, so repeat the short guidance at intervals and attach it to the places where requests actually arrive, such as the payment approval form.

Avoid publishing details that would help the next attacker. A notice that describes a lure in full may teach the same lure to a different audience. Say enough for people to protect themselves, and keep the operational specifics internal.

Practice with a tabletop scenario

Design a scenario that combines the new cases. A wave of convincing payment requests reaches finance, one of them leads to a suspicious sign-in, and an agent starts reading mailboxes it does not normally touch. Add a customer-facing assistant that begins giving wrong information. Keep the scenario generic so that it can be run without touching real systems.

Run it with a decision log. At each step, ask the team to record what they know, what they decide, who makes the decision and what evidence supports it. Introduce new information at intervals to create time pressure. Afterward, debrief on what the team lacked: a contact, a kill switch, a log field or a clear authority to act.

Judge your own plan

Take your current incident response plan and mark every step that assumes the attacker's message is poor quality, or that a person can tell a real request from a fake by reading it. Rewrite those steps around behavior and verification. Then run a short tabletop with two colleagues using the scenario above, and note where the plan sent people to a decision that nobody had authority to make. Each one is a gap to close before a real incident finds it.