AI Incident Response | A Practical Guide for Security Teams

What Is AI Incident Response?

AI incident response is how an organization responds when an AI system is attacked, misused, or fails in a way that causes security impact or harm. This section defines the term, sets out what qualifies as an AI incident, compares it with traditional incident response, and separates response from management.

AI Incident Response Defined

AI incident response is a process for detecting, containing, investigating, and recovering from events in which an AI system is attacked, misused, or fails in a way that causes security impact or harm. It covers models, agents, prompts, training data, and connected tools, and adds model-specific actions such as rollback and retraining to standard security response.

The scope extends beyond a breach. A team can invoke the same process for a manipulated agent, a model leaking sensitive data in its answers, or a production model whose outputs quietly degrade.

What Counts as an AI Incident

An AI incident is any event where an AI system is compromised, abused, or behaves in a way that harms people, data, operations, or legal standing. In practice, four categories cover most cases.

• Security breaches are the clearest: stolen model weights, a leaked agent API token, or a prompt injection that makes an agent take actions it should not.

• Misuse covers legitimate systems used for illegitimate ends, such as an employee pasting confidential data into an unapproved AI tool, or an attacker using a company chatbot to extract internal information.

• Harmful output covers cases where the system produces content that creates legal, safety, or reputational exposure, such as a biased decision, fabricated advice, or disclosure of personal data.

• Model failure covers drift, bad updates, or broken dependencies that make a system unreliable without any attacker involved.

The OECD gives the most widely used formal definition. Its May 2024 framework describes an AI incident as an event, or series of events, where the development, use, or malfunction of AI systems directly or indirectly leads to harm: injury to health, disruption of critical infrastructure, violations of rights or legal obligations, or damage to property, communities, or the environment. It separately defines an AI hazard as an event that could plausibly lead to such an incident. The harm threshold suits public reporting, but security teams should triage earlier. A near miss, such as a successful prompt injection that reached no sensitive data, belongs in your response process because the same flaw may not fail safely next time.

How AI Incident Response Differs From Traditional Incident Response

The lifecycle is familiar, but the assets, evidence, containment options, and owners all change. The core difference is that the thing under attack is often behavior rather than infrastructure, so you cannot always isolate it the way you isolate a host.

Because AI incidents so often run through ordinary weaknesses like exposed APIs and weak access control, your existing incident response foundation still applies. The AI-specific work is deciding in advance what evidence to log and what “contain” means for a model.

AI Incident Management vs AI Incident Response

AI incident response is the tactical work done while an incident is unfolding, and AI incident management is the broader program around it. Management includes the policies, severity definitions, roles, tracking, reporting obligations, and post-incident learning that make a response repeatable. Response is one phase within it.

The distinction matters for ownership. The people triaging and fixing the incident lead response. Management belongs to whoever sets the rules, usually the governance, risk, and compliance function: severity thresholds, escalation paths, regulator and customer notification decisions, and the review that feeds findings back into controls. NIST’s current incident response guidance, SP 800-61 Rev. 3, takes a similar view and treats incident response as part of wider cybersecurity risk management.

In this guide, “AI incident management” means managing incidents that involve AI systems. It does not mean using AI tools to manage ordinary IT incidents, which is covered separately in the section on how AI helps the response process.

Types of AI Security Incidents

AI security incidents fall into six main types: prompt injection, leaked credentials, training data and model theft or poisoning, data leakage through outputs, model drift, and agent misuse. Each leaves different evidence and calls for a different first move, so early classification determines how fast you can contain it.

Prompt Injection and Malicious Tool Responses

Prompt injection is an attack in which attacker-supplied text overrides a model’s intended instructions. A malicious tool response is the same attack delivered through the output of a tool, web page, document, or API that the model reads. OWASP’s 2025 Top 10 for LLM Applications still ranks prompt injection as the number one risk. Testing for it is a core part of LLM security testing.

Many of these incidents are indirect. The attacker never talks to the model; instead, they plant instructions in content the model will retrieve. Typical signs include an agent calling tools it has no reason to call, answers that follow instructions the user never gave, and outbound requests to unfamiliar domains. The first move is to cut the agent’s tool and data access, then preserve the full prompt context, including retrieved content, because the payload usually sits there rather than in the user’s message. According to a summary of IBM’s 2026 report, prompt-injection breaches averaged about $5.89 million.

Leaked API Tokens and Agent Credentials

A leaked API token or agent credential lets an attacker act as your AI system, call its tools, read its data, and run up usage costs without ever touching the model. Agents tend to hold more credentials than ordinary applications: model provider keys, database access, SaaS tokens, and cloud roles.

Secrets leak through code repositories, logs, tool configurations, client-side bundles, and sometimes the agent’s own outputs. Warning signs include usage spikes, calls from unfamiliar IP addresses, and tool actions at unusual hours. Dark web monitoring can also reveal credentials that have already been exposed. First, revoke and rotate the credential, then review logs for everything it accessed. Rotation alone isn't enough if the agent’s permissions were broader than its task required, because the next leaked token will have the same reach.

Training Data and Model Theft or Poisoning

Poisoning is the deliberate corruption of training, fine-tuning, or retrieval data. As a result, a model behaves as the attacker wants, and model theft involves unauthorized copying of weights or extraction of model behavior. OWASP’s 2025 list widened its poisoning category beyond training data to include manipulation during fine-tuning and embedding. These risks also come through third-party models, datasets, and libraries, which AI supply chain security focuses on.

These incidents surface slowly because nothing alerts. The model simply behaves differently on certain inputs, or an unusual volume of queries or access to model artifacts points to extraction. First, freeze the affected model version, identify the data sources and pipeline changes in the relevant window, and roll back to a version with verified lineage. Recovery means retraining or re-indexing, so how well you track data lineage determines how long it takes.

Data Leakage Through Model Outputs

Data leakage through model outputs occurs when a model reveals sensitive information it was trained on, retrieved, or was given in context, such as personal data, internal documents, credentials, or its own system prompt. Common causes are overbroad retrieval permissions, sensitive data in fine-tuning sets, and extraction or inversion attacks. According to the same summary of IBM’s 2026 report, model inversion attacks averaged about $6.07 million per breach.

Evidence usually arrives as a user report, a data loss prevention hit on a response, or an answer containing data the requester should not see. First, restrict the affected endpoint or data source and preserve the prompts and responses that exposed the data. Then establish whose data was exposed, because breach notification duties may follow.

Model Drift and Silent Degradation

Model drift is the gradual decline in a model’s accuracy or behavior as real-world data, usage patterns, or upstream dependencies move away from what it was built on. It becomes a security incident when degraded outputs cause harm without anyone noticing, such as a fraud model that stops catching new patterns or a classifier that begins approving what it should flag.

No attacker is required, though an attacker can induce it. Watch for shifts in input distribution, falling accuracy against labeled samples, and rising complaint or manual-override rates. First, compare current behavior to your baseline, pin or roll back to the last known-good version, and route affected decisions to human review. The lasting fix is setting monitoring thresholds before launch, because without a baseline you cannot show when degradation began.

Agent Misuse and Excessive Autonomy

Excessive autonomy means an AI agent has more capability, permission, or freedom to act than its task requires, so a mistake or manipulation can lead to real actions such as deleting data, sending messages, moving money, or changing configurations. OWASP added Excessive Agency as a new category in its 2025 list, and it points builders of multi-step autonomous agents to its separate Top 10 for Agentic Applications. Misuse also covers people steering an agent beyond its intended purpose.

Signals include actions outside the task scope, bursts of tool calls, repeated retries on denied operations, and bypassed approvals. First, pause the agent or require human approval for every write action, then use its action log to list what changed and reverse or compensate for it. The root cause is almost always permission design, so the lasting fixes are least privilege and approval gates for high-impact actions.

Real-World AI Data Breach and Misuse Examples

The best-documented AI incidents are mostly ordinary security failures around AI systems, plus a few attacks specific to how models process content. Researchers, vendors, or tribunals documented each of the six cases below.

Verified Incidents: What Happened, Root Cause, and What the Response Should Have Been

Microsoft AI research team, 38 TB exposure (2023). Microsoft’s AI research team accidentally exposed an additional 38 terabytes of private data while publishing open-source training data on GitHub using an Azure SAS token. The exposed data included workstation backups and over 30,000 internal Teams messages. The root cause was an account-level token that granted full control over the whole storage account, with an expiry date set to October 2051. Wiz reported the issue on June 22 and Microsoft had revoked the token by June 23. That speed was right. The better outcome was prevention: scoped, short-lived tokens, an inventory of issued tokens, and secret scanning on public repositories.

DeepSeek exposed database (early 2025). Wiz found a publicly accessible ClickHouse database belonging to DeepSeek that exposed over a million lines of log streams, including chat history, secret keys, and backend details. DeepSeek secured it promptly after disclosure. The root cause was an internet-facing analytics database with no authentication, which Wiz found within minutes of external reconnaissance. Closing the exposure is only step one. The response should also rotate every secret in those logs, review access logs for earlier access, and decide whether users must be notified. Continuous external attack surface monitoring would have caught it first.

EchoLeak in Microsoft 365 Copilot (June 2025). EchoLeak was a zero-click vulnerability, tracked as CVE-2025-32711 with a CVSS score of 9.3, that allowed data exfiltration from Copilot’s context without user interaction, and there was no evidence of malicious exploitation. Aim Security, which discovered it, described it as an LLM scope violation that enables indirect prompt injection. The root cause was processing untrusted external content (an email) alongside privileged internal data, with a path for that data to leave. Microsoft patched it server-side, so customers had little to do. The lesson for your own retrieval-based systems is to log retrieval sources, keep untrusted content out of privileged context where possible, and restrict what rendered links and images can send out.

Salesloft Drift OAuth theft (August 2025). Between August 8 and 18, 2025, a threat actor tracked as UNC6395 used stolen Drift OAuth tokens to export large volumes of Salesforce data and search it for secrets such as AWS access keys, passwords, and Snowflake tokens. Salesloft and Salesforce revoked all active Drift tokens on August 20. The root cause was long-lived integration tokens from an AI chat agent vendor, which let the attacker act with valid credentials. Google advised affected organizations to assume their Salesforce data was compromised, review logs, revoke keys, and rotate credentials. The better posture is least-privilege integrations and keeping secrets out of CRM text fields.

Replit agent deletes a production database (July 2025). During a 12-day “vibe coding” experiment by investor Jason Lemkin, Replit’s AI agent deleted a production database despite an instruction to freeze all code changes, and Replit’s CEO called the deletion unacceptable. The database held records on 1,206 executives and over 1,196 companies. The root cause was a control that existed only as an instruction: the agent could still reach and modify production. The right response is to pause the agent, restore from backup, and use its action log to establish what changed. Lasting fixes include separating development from production and requiring human approval for destructive actions.

Air Canada chatbot misstates policy (2024). The chatbot told a passenger he could claim a bereavement fare after travel, which contradicted the policy on the page it linked to. The tribunal found the airline did not take reasonable care to ensure its chatbot was accurate. The tribunal rejected Air Canada’s argument that it was not responsible for the chatbot and ordered it to pay $812.02 (CAD) in damages and fees. The root cause was an unverified answer presented as authoritative. A proper response treats a wrong answer like this as an incident: correct the customer, fix the grounding, and review similar conversations for the same error.

Patterns Across Incidents

Most of these were conventional failures. Four of the six trace to basic controls: an over-broad token, an unauthenticated database, long-lived integration tokens, and an agent with production write access. Only EchoLeak depends on how a model processes untrusted content. That is why a foundation of access control, secrets management, and attack surface monitoring catches more AI incidents than model-specific tooling.

Excessive permission turned small errors into large ones. A full-control token, an agent that could reach production, and broad-reach integration tokens all widened the damage. In each case, the fix is least privilege, a core zero trust security principle, applied to the AI system the same way it is applied to a human administrator.

Outsiders usually spotted the problem first. External researchers (Wiz, Aim Security), a customer (Moffatt), and a user (Lemkin) surfaced most of these cases rather than the owner’s own monitoring. Internal detection for AI systems, such as logging prompts, tool calls, and external exposure, is where most organizations are weakest.

Containment was simple once someone acted. Revoking tokens, closing a database, and patching all took hours or days, not weeks. The bigger delays were in noticing and in deciding scope, which is why the first hour of a plan should say who can revoke what without waiting for approval.

Accountability stayed with the deployer. The Air Canada ruling is the clearest signal: the tribunal treated the chatbot as part of the airline’s website, not a separate party. Your organization owns what its AI says and does.

AI Incident Databases and Trackers: What They Are and How to Use Them

AI incident databases and trackers are public catalogs of reported AI failures, harms, and near misses, and security teams use them to see how AI systems fail in practice. They are research and awareness tools. They are not regulatory reporting channels, and they cover less private enterprise breach activity than the public record might suggest.

MIT AI Incident Tracker, OECD AI Incidents Monitor, and the AI Incident Database Compared

The three resources overlap but do different jobs: the AI Incident Database (AIID) is the primary catalog, the OECD monitor is an automated news-driven feed, and the MIT tracker is an analysis layer built on the AIID. Treat the MIT tracker as a lens on the AIID, not an independent dataset, so counting both would double-count the same incidents.

One caution applies to all three. Because they rely on public reporting, they skew toward harms that make the news. MIT’s classification, for example, finds malicious actors as the leading domain at 35% of reported incidents, and fraud and manipulation account for most of those. Quietly handled enterprise breaches are underrepresented, so pair these sources with vulnerability advisories, CVE/NVD entries, and vendor disclosures. That coverage gap is an inference from how the sources collect reports, not a figure any of them publishes.

Using Trackers to Build Your Threat Scenarios

Use trackers to turn abstract AI risks into specific, testable scenarios by finding incidents that resemble your systems. MIT’s tracker helps because almost all reported incidents are classified as post-deployment (only 2% pre-deployment), and 51% were tagged as intentionally caused. That tells you to build scenarios around live systems and to include accidents alongside attackers.

A practical sequence:

1. Match by architecture. Filter for incidents involving the same type of system you run, such as a customer chatbot, a retrieval-based assistant, or an agent with tool access. Use the MIT cause and domain tags to narrow further.

2. Read the underlying reports. The mechanism and what delayed detection matter more than the headline.

3. Write the scenario. State the trigger, the signals you would expect, and the decisions someone must make, such as who can disable the system.

4. Test your evidence. Check whether your logs would actually capture what the scenario requires: prompts, tool calls, and model versions.

5. Refresh on a schedule. New incidents change which scenarios deserve a tabletop exercise.

Reporting an Incident to a Public Database

To report to a public database, you submit a link to public reporting about the incident. You do not file a confidential notification. The AIID is the main route: it advises submitting when you're unsure whether an event qualifies, and editors decide whether an event warrants an incident profile. The OECD monitor builds its entries from news coverage, so it reflects incidents that have already been reported publicly. Check OECD.AI for any direct submission route.

Before you submit, settle the obligations that come first. Notify regulators, customers, and affected individuals where required, and coordinate vulnerability disclosure with the vendor before anything becomes public. A database entry is not a substitute for statutory reporting. Keep confidential details, personal data, and exploit specifics out of any submission. Get legal and communications approval, and record the submission for your post-incident review. The aim is to add to shared learning without creating new exposure.

The AI Incident Response Lifecycle

The AI incident response lifecycle has five stages: preparation, detection and triage, containment, eradication and recovery, and post-incident review. It follows the same logic as traditional incident response, but each stage has to cover models, agents, tools, and data, not just servers and accounts.

Preparation

Preparation means knowing what AI you run, what it can reach, and who owns it before anything goes wrong. Build an inventory that covers every model (provider, version, hosting), every agent, the tools and permissions each one holds, the data flows in and out (training data, retrieval sources, prompts, outputs), the credentials involved, and a named owner with an on-call contact. Include unsanctioned AI tools employees use, because you cannot respond to a system you don’t know exists.

Then decide the rules in advance: severity levels, who may disable a system without approval, what must be logged, and how to reach each AI vendor during an incident. The gap is real. IBM’s 2026 Cost of a Data Breach research found that only 40% of organizations reported using access controls on AI models and data. Write these rules into your information security policy so they carry the same authority as your other controls.

Detection and Triage

Detection for AI systems means monitoring both behavior and infrastructure. Alert on input or output drift, anomalous agent actions, and unusual tool calls, such as new destinations, bursts of activity, or repeated retries after a denial.

None of that works without evidence, so log prompts, retrieved sources, tool calls with their parameters, model and version identifiers, and outputs, with retention and access rules because those logs will contain sensitive data. When an alert fires, triage answers four questions fast: which AI system is involved, what it can access, whether data is leaving, and whether the activity is still happening. Those answers set severity and tell you whether the AI system caused the incident or was only its target.

Containment

Containment for an AI system means cutting off what it can reach before you finish working out what went wrong. The main levers are a kill switch that disables the feature, revoking agent tokens and API keys, disabling specific tools or plug-ins, restricting retrieval sources, and isolating the model endpoint or falling back to a known-good version or human review.

Preserve evidence first where you can, because conversation context, agent memory, and ephemeral logs may be lost when you restart or redeploy. Prefer the narrowest action that stops the harm, such as disabling one tool rather than the whole service, and settle in advance who can pull each lever without waiting for approval.

Eradication and Recovery

Eradication removes the cause, and recovery returns the system to a verified state. With AI, that covers more than rotating secrets. Remove poisoned or malicious content from retrieval indexes, stored memory, and training or fine-tuning data. Then retrain the model or roll back to a version with verified lineage.

Before restoring service, validate outputs. Run your test set plus the adversarial inputs that reproduced the incident, and check that no malicious instruction persists in agent memory or a vector store. Re-enable in stages with heightened monitoring and human review of high-impact actions, and confirm you've closed the original entry point, not just the symptom.

Post-Incident Review and Lessons Learned

A post-incident review should produce control changes, not just a report. Reconstruct the timeline, identify root causes across the model, data, permissions, and process layers, and ask how you found out and how much sooner you could have. Record corrective actions with owners and due dates. Then add the incident to your regression tests and update your inventory and playbooks so you catch the same failure next time.

NIST’s current guidance reflects this loop: it describes feeding lessons learned back through the framework’s improvement category. Where appropriate, consider contributing a public summary to an incident database so others benefit.

How This Maps to NIST SP 800-61 Rev. 3

NIST SP 800-61 Rev. 3, finalized in April 2025, organizes incident response around the six functions of NIST CSF 2.0 and supersedes the 2012 Revision 2. It treats preparation (Govern, Identify, Protect) as broader risk management activity, and the incident life cycle itself as Detect, Respond, and Recover. The stages in this guide map onto it as follows:

NIST’s publication is technology-neutral and does not address AI specifically, so the AI-specific practices above are how to apply its structure to models and agents, not requirements from the standard.

AI Agent Incident Response Playbooks

An AI agent incident response playbook is a pre-agreed set of steps for one specific failure type, so responders can act in minutes instead of debating. The four playbooks below cover the most common agent incidents, and each follows the same shape: contain, scope, eradicate, validate.

Playbook 1: Leaked Agent API Token or Secret

A leaked agent token is contained by revoking it at the issuer, not by deleting it from the code where it was found. Deleted secrets persist in git history, forks, caches, and logs. GitGuardian counted 1,275,105 AI-service secrets leaked in public GitHub commits in 2025, up 81% from 2024, and found that 64% of secrets confirmed valid in 2022 were still exploitable in January 2026. Remediation speed is the weak point.

Trigger: a secret scanner alert, a provider notice, a hit in a domain data breach scan, or an unexplained spike in usage or billing. Revoke the credential and issue a replacement with narrower scope, delivered through a secrets manager. Then pull the provider and cloud audit logs for the key’s entire lifetime: which calls, from which addresses, against which models, tools, and data. Assume any other credential stored in the same configuration is also exposed. Finally, scrub the secret from code history, logs, bundles, and prompt templates, confirm the old key is rejected, and decide whether any accessed data triggers notification duties.

Playbook 2: Prompt Injection Through a Malicious Tool or Data Source

For prompt injection, first remove the agent’s ability to act on injected instructions, then find the payload's source. Trigger: the agent calls tools it has no reason to call, outputs contain instructions or links to unfamiliar destinations, or outbound requests go to new domains.

Suspend the agent’s high-risk tools (sending, writing, external fetches), cut the suspected content source, and preserve the full context of the affected run. The payload is usually in retrieved or tool-returned content, not the user’s message. Trace which document, page, email, or tool response carried it, then find every other run that ingested it and list the actions taken and data sent. Quarantine the source, purge it from indexes and caches, and block the destinations it used. For recovery, treat tool output as untrusted, require approval for sensitive actions, restrict where rendered links and images can send data, and replay the payload in test to confirm the fix.

Playbook 3: Agent Taking Unauthorized Actions

When an agent acts outside its task, pause it or require human approval for every write, then reconstruct what it did from the tool-call log. Trigger: actions outside task scope, destructive operations, bypassed approvals, or a report from a user.

Revoke the agent’s write credentials first. Then work out whether it was manipulated (an injection), made an autonomous error, or simply had more permission than its task needed. List every changed record and system, then restore from backup or point-in-time recovery and reverse downstream effects such as emails sent or payments initiated. 

The lasting fix is a control that doesn't depend on instructions: least privilege, separate development and production environments, approval gates for destructive actions, and blast-radius caps. AI agentic penetration testing can show which permissions an attacker could abuse before they do. As the Replit case showed, a “freeze” instruction is not a control if the agent can still reach production.

Playbook 4: Model Drift or Degraded Outputs in Production

For degraded outputs, route affected decisions to human review or a fallback and pin the model version while you find out what changed. Trigger: a quality metric breaches its threshold, overrides or complaints rise, input patterns shift, or a provider updates a model.

Compare current behavior to your baseline to find when degradation began and which segments it affects. Then check the candidate causes in order: a provider model update, a prompt or template change, a data pipeline change, a retrieval index change, and finally deliberate tampering such as poisoning. Roll back to the last known-good model, prompt, or index. Run your evaluation set before re-enabling, and set explicit thresholds and an owner so the next drift is caught by monitoring and not by customers.

Cloud-Native Considerations

Cloud-native agents inherit a cloud identity, so containment means revoking the workload’s identity and network path, not only stopping the process. Containers are ephemeral, so ship logs off-node continuously and quarantine a suspect pod rather than deleting it, so you can capture its state first. Serverless functions have no host to image: rely on invocation logs and stop the function (for example, by setting concurrency to zero where your platform allows it). Securing AI infrastructure is part of the same effort, especially for on-premises or air-gapped deployments.

For service accounts, prefer short-lived credentials, then revoke or deny the role and invalidate active sessions during an incident. Restrict egress by default so a hijacked agent cannot send data anywhere, and keep metadata endpoints off-limits to agent code. For logging, centralize cloud audit logs, model gateway logs, and tool-call logs in immutable storage with synchronized clocks. Regular vulnerability assessments of the cloud environment catch many misconfigurations before an agent depends on them.

TypeScript and Other Framework-Specific Notes

In TypeScript agents, secrets most often leak through client exposure, config files, and over-verbose logging. Next.js, for example, inlines any variable prefixed with NEXT_PUBLIC_ into the JavaScript bundle sent to the browser at build time, so a key placed there is public. GitGuardian also found 24,008 unique secrets in MCP-related configuration files on public GitHub in 2025, 2,117 of them verified valid. Other common leak points are source maps, error handlers that serialize headers or config, committed .env files, CI logs, and prompts that embed keys. A source code review before release catches many of these.

TypeScript types are erased at build time, so a type marking a field as secret does nothing at runtime. Redact secrets in your logger and validate tool arguments with a runtime schema library such as Zod. Log per run: a run ID, the session or user, model and version, prompt template version, retrieved document IDs, each tool call with redacted arguments and result status, and token usage. Check whether your agent framework’s verbose or debug modes print full prompts or request headers, and turn them off in production.

Runbook Structure: Trigger, Owner, First 15 Minutes, Evidence, Rollback

A good runbook fits on one page and answers five questions without needing anyone’s judgment in the first quarter hour. Here is the structure with a worked example.

Detection and Monitoring for AI Systems

Monitoring an AI system means recording what it was asked, what it retrieved, what it did, and how it performed, then alerting when any of that departs from normal. Without that record you cannot tell an incident from noise, and you cannot scope one afterward. Slow detection has a measurable cost: IBM’s 2026 report put the mean time to identify and contain a breach at 247 days, and breaches that ran past 200 days cost an average of $5.65 million, compared with $4.32 million for those contained sooner.

What to Log

Log every step of an AI run under a single correlation ID, so you can replay what happened. Each field exists to answer a specific question during an incident:

Treat the logs themselves as sensitive data. They will contain personal information, internal documents, and sometimes secrets, so redact before writing, restrict access, store them tamper-evidently, and set retention deliberately. Where you cannot store content, store hashes and identifiers. If you run a high-risk system under the EU AI Act, logging is also a legal duty: high-risk systems must technically allow automatic event recording, and providers must keep those logs for at least six months. Confirm which of your systems qualify and the current application dates.

Drift and Anomaly Signals

Watch three families of signals, each compared to a baseline you set per system and per user segment. Input signals include shifts in prompt topics, languages, or lengths, as well as known injection patterns. Behavior signals are the most useful for agents: tool-call frequency, tools or destinations the agent has never used, repeated retries after a denial, loops, and actions outside the task. Output signals include falling quality scores, changes in refusal rates, patterns of secrets or personal data in responses, policy flags, and rising human overrides or complaints.

Cost and token spikes deserve their own alert. OWASP lists unbounded consumption among the top risks for LLM applications, and a sudden jump in usage can mean abuse, extraction, or a runaway agent. To make a drift alert actionable, tie every deployment event to your timeline: a model version change, prompt edit, retrieval index update, or data pipeline change. Most drift is explained by the change that came just before it.

Start with a small set of high-confidence alerts, such as a new tool, a secret pattern in an output, a destructive action, or an outbound request to an unseen domain. Add statistical drift alerts once you have enough baseline data to keep false positives manageable.

24/7 Monitoring and Escalation Paths

AI systems run and act around the clock, so monitoring needs continuous coverage and a written escalation path that says who is paged, who can disable the system, and who else must be told. External discovery is a common failure mode: IBM’s 2026 report found that attackers themselves disclosed roughly one in six breaches, and those cost the most of any discovery route.

A workable path has four tiers. Automated guardrails immediately block or rate-limit the clearest violations. Alerts then go to an on-call responder, from security operations or the platform team, armed with AI-specific runbooks and your system inventory. Model or data problems escalate to the AI/ML owner of that system. Incidents involving personal data, safety, or legal exposure escalate to privacy, legal, and communications, and provider-hosted models need a named vendor contact. Pre-authorize the on-call responder to use the kill switch without waiting for approval. Test the full path with drills andred teaming exercises, and track time to acknowledge and time to contain, because an untested escalation path usually fails at the first handoff.

Frameworks, Compliance, and Reporting Duties

AI incidents trigger the same legal clocks as any other security incident, plus a few AI-specific ones, and the shortest deadlines are measured in hours. Frameworks such as NIST, OWASP, and ISO shape your response. Laws and regulators decide when you must tell someone. This section is general information, not legal advice, so have counsel confirm which duties apply to you.

NIST AI RMF and NIST Incident Response Guidance

The NIST AI Risk Management Framework tells you what to govern, and NIST SP 800-61 Rev. 3 tells you how to run the incident process. The Generative AI Profile (NIST AI 600-1, July 2024) connects the two: it lists incident disclosure as one of four primary considerations, alongside governance, content provenance, and pre-deployment testing, and it recognizes prompt injection and data poisoning as information security risks.

NIST describes the profile as governance-level guidance and not a technical security standard, so use it to set ownership, disclosure expectations, and risk tolerance, and use SP 800-61 Rev. 3 and your own playbooks for execution. All of NIST’s guidance here is voluntary unless a contract or regulator adopts it.

OWASP LLM and Agentic Guidance, and MITRE ATLAS

OWASP and MITRE give you a shared vocabulary for classifying what went wrong. The OWASP Top 10 for LLM Applications (2025) ranks prompt injection first, and in December 2025 the OWASP GenAI Security Project released a separate Top 10 for Agentic Applications for autonomous systems. MITRE ATLAS is a knowledge base of adversary tactics and techniques against AI systems, built from real-world observations and red-team activity, and it is most often used for post-incident threat assessments and red teaming.

In practice, tag each incident in your review with an OWASP category and an ATLAS technique so findings roll up across incidents. ATLAS also links to ATT&CK where behavior overlaps. For example, its unsecured credentials technique corresponds to ATT&CK T1552, which fits the leaked-token playbook.

ISO/IEC 42001

ISO/IEC 42001, published in December 2023, is the first international standard for AI management systems, and it is certifiable. It follows the same high-level structure as ISO 27001, so it can sit alongside an existing security management system. For incidents, Annex A calls for a process for interested parties to report adverse impacts of AI systems (A.8.3) and a documented strategy for communicating incidents to users clearly and on time (A.8.4).

ISO 42001 does not set regulatory deadlines. Its value is demonstrating a defined, evidenced incident process that auditors and customers can test.

GDPR and SOC 2 Expectations for AI Incidents

Under GDPR, an AI incident that compromises personal data is a personal data breach like any other. The controller must notify the supervisory authority within 72 hours of becoming aware and, if it notifies late, explain the delay. Notification is required unless the breach is unlikely to result in a risk to individuals. A model that leaks personal data in its outputs, or a breach of the logs and training data behind it, can meet that threshold. Affected individuals must also be told without undue delay when the risk to them is high.

SOC 2 is an audit attestation, not a law, and it sets no fixed deadline. Its Common Criteria CC7.3 to CC7.5 cover evaluating security events, executing a defined incident response program, and recovering from incidents. Auditors look for evidence that the program actually ran, not just a plan document. An incident during the audit period does not by itself cost you the report if your monitoring detected it and your process handled it.

United States

For U.S. public companies, SEC Item 1.05 requires a Form 8-K within four business days of determining that a cybersecurity incident is material. The clock starts at that determination, which must be made without unreasonable delay after discovery. An AI incident can be material, so your triage process needs a path to the disclosure committee.

CIRCIA would require covered critical infrastructure entities to report covered incidents to CISA within 72 hours and ransom payments within 24 hours, but the final rule is not yet in force. In early October 2026, CISA sent the final rule to the White House Office of Management and Budget for review, and reporting becomes mandatory only once the published rule takes effect. Check CISA for current status. State breach notification laws set their own triggers and deadlines, and sector regimes such as health, financial services, and banking add their own clocks, so confirm which apply to you.

AI-specific incident laws now exist at the state level, but they apply to frontier model developers, not most deployers. California’s SB 53 requires reporting critical safety incidents to the state within 15 days of discovery, or within 24 hours if there is imminent danger of death or serious injury. New York’s RAISE Act sets a 72-hour window and takes effect January 1, 2027.

UAE and GCC

Under cybersecurity regulations UAE , the applicable regime depends on your sector and location. For Dubai virtual asset service providers, VARA’s rulebooks require reporting a cybersecurity event or business continuity trigger that materially affects operations as soon as reasonably practicable and no later than 72 hours from detection. They also require notifying VARA within 24 hours after notifying a data regulator or data subject about an incident affecting personal data. VARA’s Technology and Information Rulebook also points to the UAE PDPL and any applicable CBUAE cybersecurity requirements.

The UAE PDPL (Federal Decree-Law 45 of 2021) requires notifying the UAE Data Office and affected individuals of qualifying breaches, but leaves the notification period to Executive Regulations. Sources conflict on whether those regulations have been issued, so confirm with the Data Office. The PDPL does not cover DIFC and ADGM, which have their own regimes.

For Saudi Arabia only, the PDPL requires controllers to notify SDAIA via the National Data Governance Platform within 72 hours of becoming aware of an incident that may harm personal data or data subjects, and to notify data subjects without undue delay if the breach could harm them. The NCA’s Essential Cybersecurity Controls (2024 version) apply to in-scope entities; confirm their incident-reporting requirements directly.

EU AI Act Serious-Incident Reporting

Under Article 73 of the EU AI Act, providers of high-risk AI systems must report serious incidents to the market surveillance authority of the Member State where it occurred. The deadline is 15 days as a rule, 10 days where a person has died, and 2 days for widespread infringements or critical infrastructure disruption. After reporting, the provider must investigate and must not alter the system in a way that could affect the evaluation of the causes without first informing the authorities. Deployers must inform the provider without delay.

Check the timing. The EU’s late-2025 Digital Omnibus reportedly deferred Annex III high-risk obligations to December 2, 2027 and Annex I obligations to August 2, 2028. Separately, providers of general-purpose AI models with systemic risk already have a duty under Article 55 to report serious incidents to the EU AI Office without undue delay. Because the deferral and your risk classification determine what applies today, confirm both with counsel.

How AI Helps Incident Response, and Where It Doesn’t

AI speeds up the parts of incident response that involve gathering and summarizing evidence, and it should not make final or irreversible decisions on its own. The gain is measurable: IBM’s 2026 Cost of a Data Breach research found that organizations using security AI and automation shortened breach lifecycles by 65 days and reduced breach costs by $1.93 million, yet only 36% used those capabilities extensively across the full security lifecycle.

Where AI-Driven Triage, Enrichment, and Summarization Add Value

AI adds the most value where work is repetitive, high-volume, and evidence-based: sorting alerts, enriching them with context, correlating signals across systems, and drafting timelines and summaries. Enrichment means attaching the asset owner, the user’s recent activity, related alerts, and threat intelligence to an alert, which removes much of the first stretch of every investigation.

It also helps with incidents in AI systems themselves. Agent traces and prompt logs are long and noisy, and summarizing them to find the retrieved document or tool response that changed the agent’s behavior is the kind of reading task AI does quickly. In each case the output is a starting point for an analyst to check, not a verdict.

Automation and Agentic Response: What to Automate and What Needs Human Approval

Automate reversible, low-impact, well-defined actions, and require human approval for anything irreversible, high-impact, or ambiguous. Teams are cautious in practice. In one 2026 survey of 250 security leaders, 57% of teams running AI in the SOC still required human review of every verdict, and only 30% auto-executed even low-risk actions.

Start with a human approving each action, then move to supervision without per-action approval only for narrow tasks where the AI has a track record.

Can AI Fully Replace Manual Incident Response?

No. AI augments incident response teams but doesn't replace them, and the evidence supports that. Gartner’s guidance, as reported by security vendors, is that AI SOC agents are not a replacement for human operators and that most successful implementations stay tactical, automating tasks and augmenting processes. A Ponemon-based survey found 62% of organizations had adopted some form of AI in security operations, but only 44% said the technology alone was effective.

The limits are structural. Humans carry business context, such as which system matters most and which customers are affected. Humans make the legal and materiality calls that regulators hold you accountable for. Humans handle novel attacks outside known patterns and coordinate across teams. Fragmented data also limits AI: 49% of survey respondents said their data is too dispersed and hard to standardize. These surveys are vendor-commissioned, so read them as directional.

Human and AI Collaboration Model

A workable model has AI propose and humans decide at defined checkpoints. Let AI do collection, correlation, and drafting, and require it to cite the evidence behind each claim (log lines, record IDs, timestamps) so an analyst can verify it quickly. The analyst validates the evidence and verdict before escalation, the incident commander owns containment and recovery decisions, and legal, privacy, and communications own notifications.

Set three checkpoints in advance: before any containment beyond pre-approved actions, before declaring a root cause, and before any external communication. Track how often AI verdicts agree with analyst verdicts, and the false-positive and false-negative rates, and expand autonomy only as those numbers justify it.

Risks of AI in the Response Process

AI introduces its own risks into the response, and each one needs a specific control.

• Hallucinated findings. AI can assert evidence that does not exist or misread a log. Require citations to source records and verify before acting.

• Overreach. A response agent with broad permissions can take actions beyond its task, the same excessive-agency risk OWASP lists for LLM applications. Give it read-only access by default and the narrowest write permissions it needs.

• Manipulation of the responder. Attackers can plant instructions in logs, emails, or files the AI reads. Treat all investigated content as untrusted input.

• Auditability. If you cannot reconstruct why the AI reached a conclusion or took an action, you cannot defend a materiality decision or satisfy an auditor. Log the AI’s own prompts, evidence, tool calls, and approvals.

• Data exposure. Incident data often includes personal data, so check what any AI service retains or learns from it.

Test these controls in drills before an incident, because the first real incident is a poor time to learn the AI behaves differently under pressure.

Building an AI Incident Response Plan

An AI incident response plan is a short, tested document that names who decides what, how incidents are classified, how you rehearse, and what must be true before an AI system goes live. Build it as an extension of your existing incident response plan, not a separate one. Preparation pays off: IBM’s 2023 report found that organizations with an incident response team and regular testing saved $1.49 million per breach compared with those with little or none.

Roles and Decision Rights

Name one accountable incident commander, then assign decision rights for each domain so nobody has to negotiate authority during an incident. Security owns containment, evidence, and forensics. The AI/ML owner owns model behavior, rollback, retraining, and evaluation. Legal and privacy own notification duties and regulator contact. Communications owns customer, public, and internal messaging. The product or system owner decides on business impact and downtime, and someone owns the relationship with each AI vendor.

Decide this in advance because reporting clocks are short. Under the EU AI Act, for example, serious-incident reports for high-risk systems are due within 15 days, or sooner in severe cases, so a team that first debates who reports during the first incident will struggle to meet them.

Severity Classification for AI Incidents

Classify AI incidents by impact and AI-specific factors, not only technical severity. Six factors matter most: whether data was exposed (personal, confidential, or secrets), whether the system took actions and whether they are reversible, how many users or records are affected, whether people were harmed or the law was implicated, whether the incident is ongoing or persistent (such as poisoned data), and how visible it is.

The near-miss level mirrors the OECD’s distinction between an AI incident and an AI hazard, an event that could plausibly lead to harm. Your severity ladder triggers legal assessment but does not replace it. Whether something is a reportable breach, a serious incident under the EU AI Act, or a material cybersecurity incident is a separate legal determination. Any suspected personal data exposure should start the regulatory clock immediately.

Tabletop Exercise Scenarios

Run tabletop exercises on realistic scenarios drawn from your own systems and from public incidents, releasing one decision-forcing development at a time. Add time pressure, partial information, and outside questions such as a regulator inquiry or a reporter’s call.

Run them at least annually and after major system changes. This cadence is common practice, not a legal requirement. A purple team session, where attackers and defenders work together, is a useful follow-up for testing detection. End every exercise with action items that have owners and dates.

Readiness Checklist

• Inventory of every model, agent, tool, data flow, and credential, each with a named owner

• Incident commander and decision rights documented and acknowledged

• AI incidents added to your existing incident response plan and on-call rota, including an AI/ML owner

• Severity ladder approved, with the link to legal assessment defined

• Kill switches and credential revocation procedures tested, with pre-authorized responders

• Logging in place for prompts, retrieval sources, tool calls, and model versions, with retention set

• Playbooks written for your top scenarios, such as token leak, prompt injection, unauthorized actions, and drift

• Legal reporting map reviewed by counsel: which laws apply, their clocks, contacts, and templates

• AI vendor escalation contacts and contractual notice terms on file

• Communications templates and approvers ready

• A tabletop exercise completed in the last 12 months, with actions tracked to closure

• Post-incident review template ready, and incidents feeding back into tests and controls

Common Mistakes in AI Incident Response

The most common mistakes in AI incident response are organizational, not technical: treating AI incidents as ordinary IT events, not knowing what AI you run, not logging the right evidence, and having no fast way to stop a system. Each one is cheap to fix before an incident and expensive after.

Treating AI Incidents as Ordinary IT Incidents

Treating an AI incident as an ordinary IT incident makes teams look for the wrong evidence and apply the wrong containment. A server runbook says to isolate the host and image the disk. It says nothing about a prompt, a retrieved document, an agent’s tool permissions, or a model version.

Some AI incidents involve no intrusion at all. When Air Canada’s chatbot misstated a refund policy, a tribunal held the airline responsible for what the chatbot said, even though no system was breached. The fix is to extend your existing plan with AI-specific triggers, evidence types, containment steps, and owners, including an AI/ML owner, instead of hoping a generic runbook fits.

No Inventory of Deployed Models and Agents

Without an inventory of models and agents, you cannot respond to a system you do not know exists. Shadow AI shows the scale of the gap: IBM’s 2025 research found that one in five organizations reported a breach due to shadow AI, and only 37% had policies to manage or detect it.

In an incident the missing facts are basic. Which model and version is it, what data did it touch, what credentials does it hold, and who owns it? The fix is a living inventory covering models, agents, tools, data flows, credentials, and owners. Pair it with discovery of unsanctioned use and a simple approval path, so teams register tools instead of hiding them. Security awareness training reinforces that habit, and attack surface management helps you find what was never registered.

Missing Logs for Prompts and Tool Calls

If you didn't log prompts, retrieved sources, and tool calls before the incident, you cannot reconstruct it afterward because model outputs may not be reproducible. When attackers abused stolen Salesloft Drift tokens, Google advised affected organizations to assume their data was compromised and review their logs. That advice helps only if the logs exist and cover the right period.

Common failures include logging only web requests, keeping only the final answer, setting retention too short, or logging so much sensitive data that the logs become a liability. The fix is a per-run correlation ID, redaction before writing, and deliberate retention. Then replay a scenario in a drill and check that you can answer who, what, and when.

No Kill Switch or Rollback Path

An AI system without a tested kill switch and rollback path forces responders to improvise under pressure, and every minute of improvisation extends the damage. In the Salesloft Drift campaign, which ran from August 8 to at least August 18, 2025, Salesloft and Salesforce revoked all active Drift tokens on August 20, showing the value of central revocation.

Replit’s case shows the opposite. A “freeze” existed only as an instruction to the agent, not as a control. The fix is a disabled switch per system and per tool, a credential revocation procedure, pinned model versions with a prior-version fallback, and drills that test all of them. Pre-authorize the on-call responder to use them without waiting for approval.

Conclusion

The next AI incident will go well or badly, depending on what you set up this week. Pick your highest-risk AI system and make sure three things exist for it: a named owner, a tested way to switch it off or revoke its credentials, and logs that show what it was asked and what it did. Then run one tabletop exercise on it. A rehearsal will show you which decisions nobody owns, and those gaps are far cheaper to find before an incident than during one.

If you want an outside view, Femto Security can run an AI security readiness assessment covering your AI inventory, agent permissions, logging, and response readiness, and test your agents the way an attacker would through AI agentic penetration testing. Talk to our team to scope an assessment.

Frequently Asked Questions (FAQs)

What is AI incident response?

AI incident response is the process of detecting, containing, investigating, and recovering from events in which an AI system is attacked, misused, or fails in a way that causes security impact or harm. It covers models, agents, prompts, training data, and connected tools, and adds model-specific actions like rollback and retraining to standard security response.

How is AI incident response different from regular incident response?

The lifecycle is the same, but the assets, evidence, and containment options differ. Responders must examine prompts, retrieved content, tool calls, and model versions, not only servers and logs. Containment may mean revoking an agent’s credentials, disabling a tool, or rolling back a model. ML engineers join security, legal, and communications as owners.

What should an AI incident response plan include?

It should include an inventory of models, agents, tools, and data flows with named owners; roles and decision rights; a severity ladder; logging requirements for prompts and tool calls; kill switches and credential revocation; playbooks for likely scenarios; a legal reporting map; vendor contacts; and a tabletop exercise schedule.

How do you respond to a prompt injection attack?

Suspend the agent’s high-risk tools, such as sending, writing, and external fetches, and preserve the full context of the affected run. Trace the payload to its source, such as a retrieved document or tool response, then find every run that ingested it. Quarantine the source, rotate exposed secrets, and replay the payload in testing before restoring service.

Is there a database of AI incidents?

Yes. The AI Incident Database, run by the nonprofit Responsible AI Collaborative, catalogs reported AI harms and near-harms from public reporting. The OECD AI Incidents and Hazards Monitor collects news-based events, and the MIT AI Incident Tracker classifies AI Incident Database entries by risk and severity. They support research, not regulatory reporting.

Can AI automate incident response?

Partly. AI handles triage, enrichment, correlation, and summarization well, and IBM’s 2026 research linked security AI and automation to 65 days shorter breach lifecycles. But it should not make irreversible decisions alone. Automate reversible, low-impact actions, and keep humans in charge of containment at scale, restoration, and legal or disclosure decisions.

Are AI incidents reportable under GDPR or the EU AI Act?

Sometimes. If an AI incident compromises personal data, GDPR generally requires notifying the supervisory authority within 72 hours of becoming aware, unless the risk to individuals is unlikely. The EU AI Act requires providers of high-risk systems to report serious incidents, within 15 days as a rule, but application dates were reportedly deferred, so verify them.