AI readiness assessment: how to run one that changes decisions

What an AI readiness assessment is actually for

Most AI readiness assessments produce a colourful maturity chart and no decision. The purpose of the exercise is narrower than that: to work out which AI use case this organisation can put into production safely within one quarter, what must be fixed before it starts, and what would cause you to stop.

That framing matters because readiness is not a general property of an organisation. A business can be entirely ready to deploy meeting summarisation and completely unready to let a model touch client records. Assess the use case and the organisation together, or the score means nothing.

The practical test of a good assessment is whether an executive can read the output and approve or refuse spend on the strength of it. If the answer needs another workshop, the assessment failed.

The six dimensions worth scoring

Business value. Is there a named process, a measured baseline (hours, cycle time, error rate, cost) and a person accountable for the improvement? An unmeasured baseline is the single most common reason AI benefits cannot be proved later.

Data readiness. Where does the information live, who already has access to it, how accurate is it, and how sensitive is it? For most Australian organisations the binding constraint is not model quality — it is that the SharePoint or file share the model would read is over-permissioned.

Process readiness. Is the workflow stable and documented enough to automate, or does it vary by person? Automating an unstable process encodes the variation.

Security readiness. Identity and access hygiene, logging, tenancy and data residency, and whether staff are already pasting company information into consumer tools. The ACSC guidance on engaging with artificial intelligence is the right baseline reference here.

Governance readiness. Is there an acceptable use position, a way to record an AI system in a register, a human-oversight expectation for consequential outputs, and a route to assess privacy impact? ISO/IEC 42001 and the NIST AI RMF both describe this without requiring a certification programme to get value from it.

Delivery capacity. Who will configure, test, train, support and review this after the pilot? Capacity is the dimension organisations score most generously and regret most often.

Score against evidence, and let the weakest dimension rule

Score each dimension one to five, and require an artefact for anything above a three: the access review report, the process map, the baseline measurement, the register entry. A score without evidence is a preference.

Do not average. Compute the overall position as the lowest dimension, because that is how it behaves in practice — strong value and strong capacity do not compensate for unmanaged data access. A use case scoring 5, 5, 4, 2, 4, 4 is a two until the security gap is closed, and stating it that way makes the remediation fundable.

Where a dimension scores two or below, write the specific gap, the owner and the earliest realistic close date rather than a generic recommendation. "Rationalise SharePoint permissions" is not actionable. "Remove organisation-wide sharing links from the three sites the pilot will index — IT Ops, by 30 October" is.

The Australian obligations that shape the score

Privacy Act obligations apply to personal information regardless of whether a model is involved, and the OAIC has been explicit that using personal information to train or prompt an AI system is a use that must be within reasonable expectations and covered by your notices. Anything indexing customer, employee or health data needs a privacy assessment before the pilot, not after.

Sector overlays change the threshold. APRA-regulated entities carry CPS 234 information asset obligations that extend to AI services managed by third parties. Critical infrastructure entities carry SOCI risk management programme duties that name supply chain and personnel hazards. For everyone else, the Essential Eight remains the practical security floor an insurer or enterprise customer will ask about.

None of these prevent AI adoption. They determine which use case goes first, and they are the reason a readiness assessment written for a US template usually understates the work.

Running the assessment in a fortnight

Days one to three — collect. Pull the identity provider report on external and privileged access, the list of AI tools already appearing in expenditure and network logs, and the two or three candidate processes with their current volumes. Shadow AI discovery almost always changes the priority order.

Days four to seven — interview and score. Two hours with the process owner, one hour with whoever administers the tenancy, one hour with the person who will support it. Score in the room, with the evidence on screen.

Days eight to ten — decide. Pick one bounded use case, write the gaps that must close before it starts, set the measures (the baseline, the target, the review date) and write the stop condition explicitly: what result or incident would end the pilot.

The output is one page of decision plus one page of gaps. Anything longer will not be read by the person whose approval you need.

What the finished assessment should contain

A readiness score per dimension with the evidence cited. A single recommended use case with its owner, baseline and target. A gap list with dates. A named human-oversight arrangement for any output that affects a person. A register entry for the AI system, even if the register is currently a spreadsheet. And a stop decision the sponsor has agreed in advance.

Organisations that do this find the pilot is rarely the hard part. The hard part is that the assessment surfaces two or three access and data-quality problems that predate AI entirely — and fixing those is what makes the second and third use case cheap.

Common ways the assessment goes wrong

Assessing the organisation instead of a use case, which produces a maturity level nobody can act on. Scoring on opinion, which produces optimism. Averaging dimensions, which hides the blocker. Choosing a use case with no measurable baseline, which makes the benefit unprovable at review. And running the assessment without the person who will operate the result, which reliably produces a pilot with no owner in month three.

The correction for all five is the same discipline: evidence behind every score, one use case, the lowest dimension governs, and a named owner before anyone signs a licence.