The AI readiness audit

Most “AI readiness assessments” are sales tools dressed as diagnostics. The reading is shaped to end at “you’re ready” because the firm running the assessment also wants to sell you the build. We’ve sat on the other side of those meetings. They are not honest.

A real readiness audit ends, sometimes, in “not yet.” The version below is the eight-question diagnostic we walk through on every pre-engagement call. About half of the questions have nothing to do with AI. That is the point.

The eight diagnostics

I. Is there a workflow owner who’d run this after we leave?

This is the first question for a reason. If the answer is no, nothing downstream matters. We’ve watched three otherwise-perfect AI builds fail because the engineer who shepherded the project left for a higher-paying role two months after handoff. There was no second owner. The system kept running for a while, then it didn’t, and no one inside the company knew enough to fix it.

The owner doesn’t need to be a senior AI engineer. They need to be the person whose Monday-morning task involves looking at the output and noticing if something is off. If you can name them, the rest of the audit is worth running. If you can’t, fix that first.

II. Has the workflow converged enough to be specified?

Workflows that are still being figured out by humans cannot be automated. Not because the model can’t handle ambiguity — models handle some ambiguity well — but because you have no way to measure whether the model got it right.

The diagnostic question we ask: “If two of your most experienced people did this workflow on the same input, would they produce the same output?” If the answer is “mostly,” you have room to automate. If the answer is “they’d argue about which version is right,” the workflow hasn’t converged. Run it through humans for another quarter. Codify the rubric. Then come back.

III. Is the data clean enough for retrieval to find a needle?

For any workflow that involves looking things up — knowledge assistants, customer support routing, contract triage, anything with a retrieval layer — the bottleneck is usually not the model. It’s the index. We see the same pattern repeatedly: teams bolt a smarter model onto an index that was never built to be answered from, then wonder why the answers are confidently wrong.

The check: take 20 questions you would expect a customer or operator to ask. For each one, can a human find the answer in your data sources within 30 seconds? If 30 percent of those questions don’t have a clean answer in your data, no model will rescue the build. The retrieval work comes first.

IV. Is human-in-the-loop discipline already in place for non-AI work?

AI workflows almost always need a human review step in the first six months — sometimes longer. If the team doesn’t already have a review queue for non-AI work, the AI build adds a queue the team doesn’t know how to operate.

We’ve seen this fail in legal review automation. The AI was proposing contract redlines accurately. The team had no existing process for reviewing AI-generated redlines, so the redlines piled up unread. After two weeks, the legal lead silently turned the system off. The model worked. The workflow didn’t.

V. Is the engineering team set up to ship to production reliably?

This one trips up firms that don’t usually run software in production. If your engineering rhythm involves manual deploys, no monitoring of background jobs, and the only person who can restart a service is on PTO half the year, the AI workflow inherits all of those constraints.

We’d rather know now. The fix isn’t to skip the AI build — it’s to scope it inside what the team can actually operate. Sometimes that means batch processing instead of real-time. Sometimes it means a Notion table instead of an in-product surface. Both are fine. The point is to match the build to the operating capacity.

VI. Are success metrics agreed and instrumented?

Half the audit conversations we have stall here. Teams know what they want — “answer customer questions faster” — but they have no instrumentation that would tell them whether the answer was faster, or even whether the customer asked the question in the first place.

The diagnostic: ask the team to write down the metric they’d use to declare the workflow successful, and the dashboard they’d use to track it. If the metric exists but the dashboard doesn’t, build the dashboard before the AI. Two weeks of instrumentation work saves six months of post-launch arguments about whether the system is working.

VII. Is the team model-agnostic, or locked into one vendor?

A team with a hard commitment to one model provider — usually because of an enterprise discount or a procurement preference — will struggle when that provider’s model degrades, or a better one ships, or pricing changes. Model-swap discipline is what makes a workflow last more than 18 months.

We don’t require teams to be vendor-agnostic on day one. We do require them to keep the prompt-and-eval layer separate from the model call, so a swap is a configuration change rather than a rewrite. If the codebase is locked to one provider’s SDK in every file, we flag that as technical debt to retire before the first model swap, not after.

VIII. Is leadership ready to be told “not yet”?

This is the question that decides whether the engagement is even possible. If the executive sponsor has already told the board “we’re shipping AI in Q3,” the audit is no longer an audit — it’s a rationalization exercise. We’ll be asked to find ways to make Q3 work, regardless of what the diagnostics say.

We’ve declined three engagements this year on this question alone. The work would have shipped. It would have been wrong. The board would have asked questions in two quarters and the team would have been stuck explaining a system they didn’t want to build but couldn’t back out of. Better to refuse the engagement than be the firm that built the regrettable thing.

Why half of these have nothing to do with AI

Five of the eight diagnostics are about the foundation under the AI: ownership, workflow convergence, data, human review, ship discipline. The other three are about AI-specific posture: instrumentation, model-agnosticism, and leadership readiness for bad news.

We weight them roughly equally. A team that scores high on AI-specific posture but low on foundation will produce a working demo and a stalled production system. A team that scores high on foundation but low on AI posture will ship something they can’t maintain. The build needs both.

What “not yet” actually means

We refused 12 of 41 briefs last quarter. The categories were consistent. About half of those refusals were for workflow convergence (the work isn’t specified yet). A third were for ownership (no one will run it after we leave). The rest were for leadership posture (the engagement was already pre-decided).

None of those refusals were because the model couldn’t do the work. The model was almost always capable. The conditions around the model were not.

This is the read we keep giving on audit calls: the question isn’t whether AI can do your workflow. It can, or it can’t, and that’s usually obvious within an hour. The question is whether the rest of the system is ready to receive what AI produces. If it isn’t, the build is theater. Ship the foundation first.

What to do with this

Run the eight diagnostics on the workflow you’re considering automating. Score each one as green, amber, or red. If you have two reds or four ambers, the audit answer is “not yet” — and the most useful next step is the foundation work, not a vendor selection.

If you score green or single-amber across the board, you’re ready to scope a real build. The next chapter (how to pick which workflow to automate first) walks through how to rank multiple candidates against each other.