AI incident response: planning for when the model fails

Why the existing plan is not enough

Nearly every plan we review is built around intrusion: someone gets in, data leaves or systems are encrypted, and the response runs detect, contain, eradicate, recover, notify. That plan is still necessary, and it maps cleanly to NIST SP 800-61 and ACSC guidance. Keep it.

What it does not name is the way AI fails. A model gives a customer confidently wrong advice. An agent does something nobody authorised. A retrieval index hands a document to the one person who should never have seen it. No attacker involved, and not one of those trips an alert written for intrusion.

The AI incident categories to plan for

Harmful or materially incorrect output — the model gives advice, a price, an eligibility answer or a clinical or legal statement that is wrong and acted upon. Consequence scales with who received it and whether it can be retracted.

Data leakage — sensitive information leaves the intended boundary through a prompt, a retrieval index with the wrong permissions, a log that captures personal information, or a vendor feature that retains content for training.

Prompt injection and manipulation — untrusted content in an email, document or web page changes the system's behaviour, causing it to exfiltrate data, follow attacker instructions or bypass its guardrails. The OWASP Top 10 for LLM applications is the working reference here.

Excessive agency — an agent with real permissions sends messages, changes records, moves money or triggers workflows beyond its intended scope, often because the integration granted broader access than the task required.

Availability and dependency failure — the model or provider degrades, changes version, or is withdrawn, and a business process built on it stops.

Detection: what tells you something went wrong

Telemetry helps, but in practice the first signal is a person. A customer disputes an answer. Someone opens a document they should not be able to see. A manager spots an action nobody remembers approving. Give those people one obvious, blameless place to raise it.

Add machine signals where they exist: anomaly detection on agent action volumes, guardrail and refusal rates, retrieval hits on restricted classifications, sudden shifts in output length or sentiment, and unexpected outbound calls from an agent host. Baseline these while the system is behaving normally, or you will have nothing to compare against.

Containment: the switch you must be able to reach

Every AI system in production needs a documented, tested way to be turned off or reduced. That means a feature flag or kill switch, a fallback to a human or deterministic process, revocation of the agent's credentials and connectors, and the ability to disable a single integration without shutting down the whole platform.

Decide the containment tiers in advance: pause outbound actions but keep read-only assistance; restrict the system to internal users; revert to the previous model version; or full stop with human handover. Naming the tiers before the incident is what makes the decision fast during one.

Then test it. An untested kill switch is an assumption, and 9pm on a Friday is a poor time to learn that disabling the agent also takes the ticketing queue down with it.

Roles: who decides what

AI incidents cut across functions that do not usually respond together. The technology owner assesses and contains. The business owner of the affected process decides whether operations continue manually. Legal and privacy assess notification obligations. Communications handles affected customers. A single accountable decision-maker — often a fractional CIO or equivalent executive — decides whether the system resumes.

Write these into the plan by role, not by name, and include the after-hours path. Also record who is authorised to approve resumption, and on what evidence. Without that, systems either stay off far longer than necessary or come back before the cause is understood.

Investigation and the evidence you will wish you had

Reconstructing an AI incident requires more than application logs. You need the input the system received, the retrieved context it used, the model and version, the system prompt and configuration in force, the output produced, any action taken, and the identity of the human who reviewed or accepted it.

Log these at the time of use, with retention aligned to your obligations and with personal information handled appropriately in the log store itself. If the answer to "which model version produced this?" is unknown, both remediation and the regulator conversation become considerably harder.

Notification: the Australian obligations

Where an AI incident involves unauthorised access to or disclosure of personal information and serious harm is likely, the Notifiable Data Breaches scheme under the Privacy Act applies, with assessment expected promptly and notification to the OAIC and affected individuals where the threshold is met.

Sector obligations sit on top: APRA CPS 234 notification timeframes for regulated entities, SOCI Act obligations for critical infrastructure, and contractual notification clauses with clients that are often tighter than any statute. Map these into the plan so the clock starts against a known list rather than a scramble.

A harmful-output incident with no data disclosure may trigger none of these and still require customer remediation and disclosure. Decide the threshold for voluntary disclosure before you are under pressure to argue it.

Recovery and correcting the record

Recovery for an AI incident includes something intrusion response rarely does: correcting decisions already made. Identify who received the flawed output, what they did as a result, and how each of those outcomes is reversed or remediated. That work is usually larger than the technical fix.

Only then return the system to service, with the specific control that failed strengthened — tighter retrieval permissions, narrower agent scope, an added human checkpoint, input sanitisation for untrusted content, or a changed model configuration — and with monitoring focused on the failure mode you just experienced.

Rehearse it before you need it

Run an AI-specific tabletop at least annually. Useful scenarios: an agent emails confidential pricing to the wrong distribution list; a customer-facing assistant gives incorrect eligibility advice for three weeks before anyone notices; a supplier discloses that prompts were retained and used for training.

Score the exercise on the things that fail in real incidents — time to detect, time to contain, whether the kill switch worked, whether anyone knew the notification clock had started, and whether the evidence existed to reconstruct events. Feed the gaps into the plan and into the AI management system's corrective action process under ISO/IEC 42001.

A 30-day starting point

Week 1 — list every AI system in production, its owner, the data it touches, the actions it can take and whether a kill switch exists today.

Week 2 — extend the existing incident response plan with the five AI categories, containment tiers, role assignments and the notification map, rather than writing a separate document nobody will find.

Week 3 — close the evidence gaps: enable input, context, version, output and acceptance logging on the highest-consequence system first.

Week 4 — run one tabletop with the real decision-makers in the room, test one kill switch for real in a controlled window, and record the corrective actions with owners and dates.