AI Agent Incident Response: A Practical Runbook

An AI agent incident differs from a server incident in one way that changes the runbook: the agent will describe what it did and what can be recovered, and that description is not reliable evidence. This post gives a containment sequence and an evidence record, and spends its time on the points that decide whether the response works: stop controls that live in code, claims checked against tool logs, and restarts gated on evidence.

A freeze in the prompt is a request

In July 2025 an AI coding agent on Replit deleted a production database during a declared code freeze. Heise reports that the project's replit.md said no changes without explicit permission, yet the agent ran a database push command anyway. It then told the user that rollback was impossible and that no automatic backups existed. Heise says that was wrong, a backup existed and could be imported. Replit had used one database for preview, testing and production, and its CEO announced automatic separation of development and production databases afterwards.

Two rules follow for your runbook. Every stop control must be enforced by the layer that executes tools, not requested in the prompt. And during triage, treat the agent's statements about its own actions and the state of the world as claims to verify against logs and the system of record. A response plan that begins with "ask the agent what happened" starts from the least reliable source.

Build stop controls at several scopes

A single global kill switch interrupts unaffected workflows and leaves partial transactions. Expose smaller ones: pause new run admission, deny one tool, disable one workflow version, suspend one tenant, block one outbound destination, and terminate one run. Give admins per-service read and write switches and a per-tool switch, enforced in one call made by the agent loop right before a tool runs, so scheduled and unattended turns are covered too. The model should receive a clear "your team switched this off" error instead of a silent no-op. Let users stop a thread as well: a "stop" message mutes it, with no reply and no model calls after it. Build the switches before you need them. Google Cloud's report on its June 2025 outage says the faulty policy data replicated globally within seconds, and that the change was not feature-flag protected. Engineers found the root cause within about ten minutes, yet the red-button for that serving path took about 40 minutes to roll out. Recovery then hit a second failure, because restarting tasks overloaded the database and the service lacked randomized exponential backoff (Google Cloud incident report).

Record the exact time and the control you applied. Pausing the orchestrator does not prove that a queued email, payment or account change stopped, so confirm with the target system.

Check what the agent says it did

Agents also misreport their own results. In The Register's account of the Replit incident, the founder said the agent created a database of about 4,000 fictional people and fake reports, and that he had told it eleven times not to make such changes (The Register). A guard can compare claims in the reply with the tool results of that turn. A reply that says an email was sent or a ticket was solved is rejected unless the matching tool call returned success. Send the model back once with the correction, and if it repeats the false claim, rewrite the reply to say that nothing changed. A per-turn check cannot catch a claim that depends on earlier turns, so write that gap down.

For incident response this changes triage. Build the timeline from tool-call logs and downstream records, then compare it with what the agent reported to users. Mismatches in either direction are findings: actions taken that nobody was told about, and actions reported that never happened. Keep this reconciliation as a standing step in the runbook, since users act on the agent's claims.

Five ordered containment decisions

1. Stop new harm without destroying evidence. Pause admission first, then deny the consequential tool or require approval. Terminate individual runs only if they are still causing damage. If a harmful action is in progress, terminate first and preserve afterwards.
2. Preserve the decision chain. Link initiating subject, tenant, input reference, retrieved context IDs, prompt and workflow versions, model request ID, normalized tool calls, policy decision and downstream record. The OpenTelemetry agent span conventions give a vocabulary. Store references and hashes before copying raw prompts, which may hold secrets or personal data.
3. Revoke only the exposed authority. Map the suspected path to the exact session, token, key or lease before rotating anything. Broad rotation can cause a second incident. The OWASP secrets management cheat sheet treats revocation, rotation and auditing as linked controls.
4. Quarantine state derived from the suspect source. Move memory entries, embeddings, summaries, cached results and checkpoints to a quarantined namespace with provenance, not into the trash. Test whether the behavior reproduces from a clean session with the item excluded, then test the item in isolation with external tools disabled.
5. Gate the restart on evidence. The input is blocked, exposed credentials are replaced, affected records are reconciled, the vulnerable version cannot receive traffic, denied-action tests fail closed, allowed cases still work, and an owner accepts the remaining risk.

{
  "incident_id": "inc_204",
  "workflow_run_id": "run_913",
  "trace_id": "trace_551",
  "subject_id": "employee_42",
  "agent_version": "invoice-agent-18",
  "policy_version": "tool-policy-12",
  "suspected_input_ref": "document_818",
  "tool_action": "invoice.update_bank_details",
  "agent_reported": "bank details unchanged",
  "system_of_record": "bank details changed at 13:12:40Z",
  "containment": "tool-denied-and-admission-paused",
  "downstream_status": "reconciliation-required"
}

The agent_reported and system_of_record fields keep the agent's claim next to the verified state. Roles still need names: an incident commander, a platform responder who controls admission and deployments, a security responder who owns evidence and identity scope, and a business-system owner who verifies downstream records and decides on reversals. NIST SP 800-61 Rev. 3 folds incident response into risk management, and the OWASP agentic threats guide and MITRE ATLAS help scope the attack.

Keep a quiet failure channel

Operators need to learn about agent failures without users being flooded. Report failures as private issues in your tracker instead of extra chat messages, redact secrets before filing, and silence the expected kinds, such as a user who has not connected an account. The reporter should swallow its own errors by design, because a failure reporter must never cause a failure. Each issue becomes a pointer into the evidence chain without exposing anything in the channel.

Rehearse with outcomes

Run a tabletop with a synthetic prompt injection, a canary credential and a reversible tool action. The platform responder should pause admission and deny the tool without a code change. Try the negative paths: an alternate tool route, another tenant, a run already in flight when admission stops, stale memory, a redirected destination. Measure time to stop new writes, share of affected runs identified, unexplained downstream actions, and time to revoke the exposed authority. Start with the one tool that has the largest irreversible effect and write its trigger, stop control, evidence chain, credential map and reconciliation owner before the next release.

References

• heise: Replit deletes production database
• NIST SP 800-61 Rev. 3
• OWASP Agentic AI threats and mitigations
• MITRE ATLAS
• OpenTelemetry generative AI agent spans
• OWASP Secrets Management Cheat Sheet
• Google Cloud incident report, 12 June 2025: global replication within seconds, no feature flag, and a red-button that took about 40 minutes to roll out.
• The Register: Replit and the SaaStr database deletion: fabricated data and reports, and repeated instructions ignored.

---