AI risk assessment: how to assess an AI system before it ships
A repeatable AI risk assessment covering purpose, data, model behaviour, integration, human oversight, failure modes and evidence.
Start with purpose and consequence
Write down what the system decides or produces, who relies on it, and what happens if it is wrong, unavailable or manipulated. That statement determines everything that follows, including how much assessment effort is proportionate.
Distinguish between systems that inform a human, systems that draft work a human approves, and systems that act. Each step up in autonomy raises the control bar significantly.
Assess the data path end to end
Trace inputs, retrieved context, prompts, outputs and logs. Identify personal information, health information, contractual confidential data and anything subject to a sovereignty requirement, and confirm where each element is processed and retained.
Confirm the lawful basis and notice position for personal information, and whether the use is within reasonable expectations of the individuals concerned. This is where Australian privacy obligations bite hardest for AI projects.
Assess model behaviour and failure modes
Test for the failure modes that matter for your use case: fabricated content presented confidently, prompt injection through untrusted content, sensitive disclosure through retrieval, biased or inconsistent treatment of comparable cases, and degradation after a model version change.
Record the evaluation method and results rather than a general assurance. A short, documented test set that reflects your real inputs is worth more than a vendor benchmark.
Want this assessed against your environment?
Send us the specifics and a senior advisor will respond within one business day.
Assess integration and abuse paths
List every system the AI can read from or write to, the credentials it holds, and the trust level of the content it ingests. Any path where untrusted content reaches a model that can act is the highest-priority control point.
Consider misuse by legitimate users as well as external attack. Access to a capable assistant with broad data reach is an insider-risk consideration.
End with a decision, conditions and review date
A useful assessment concludes with: approved, approved with conditions, or not approved — plus the named owner, the conditions to meet, the monitoring in place and the date of the next review.
Keep the record. When a regulator, customer or board asks how the decision was made, the assessment is the answer.
Sources and further reading
- NIST AI Risk Management Framework
- ISO/IEC 42001 AI management systems
- OAIC guidance on privacy and the Privacy Act
- OWASP Top 10 for Large Language Model Applications
This article provides general information and decision support. It is not legal advice, audit assurance, certification advice or a guarantee of outcome.
Related reading
Securing enterprise AI adoption: a practical AI security control set
The AI security controls that matter first — identity, data boundaries, model access, logging, human oversight and supplier assurance.
Read articleISO 42001 vs ISO 27001: how the two management systems interlock
What each standard covers, where they overlap, and how to run one integrated management system instead of two parallel programmes.
Read articleThe AI governance operating model: roles, gates and evidence
How to design an AI governance framework that approves useful work quickly and stops harmful work early, without a committee bottleneck.
Read article