Why most AI POCs don't ship to production

We’ve picked up four AI projects in the last six months that had a working POC and no production system. Same pattern each time. The demo worked. The path to production didn’t exist.

This is the pattern we keep seeing across teams of every size: the proof-of-concept is treated as the engineering work, and the production cutover is treated as a follow-on task that “just needs an engineer.” The follow-on never gets prioritized. Six months later the POC is a Notion doc and the workflow is still running on humans.

The cause isn’t a single mistake. It’s four separate failure modes, usually present in some combination. Below is what we look for, and the day-60 ship cycle we use to close the gap.

Failure mode I — The demo data is clean. Production data isn’t.

POCs are built against a small, hand-curated dataset. Often a single CSV exported by an analyst, often with the rows that wouldn’t demo well already removed. The model performs beautifully on this dataset because the dataset has been unconsciously selected to be the kind of data the model handles well.

Production data is messier. There are nulls. There are records from systems that have been deprecated and partially migrated. There are encoding issues. There are duplicates the analyst removed before showing the team. The model that hit 92% accuracy on the demo dataset hits 64% on production traffic, and the production cutover stalls indefinitely.

The fix is to never build the POC against the curated dataset. Build it against a sample of real production data — including the records the analyst would have removed. The accuracy number will be lower. The number will also be honest, and the cutover will be possible.

Failure mode II — The eval harness was retrofitted (or never built)

A POC without an eval harness is a vibe check. The team runs a few prompts, looks at the outputs, says “that looks right,” and moves on. There is no way to know if a prompt change made things better or worse, no way to know if a model upgrade is safe, no way to know if production traffic is regressing.

Retrofitting an eval harness after the POC is built is harder than building it alongside. By the time the POC is “done,” the prompts have been edited dozens of times based on impressions. There is no test set. The team has no agreed scoring rubric. Building the harness now means going back and codifying decisions that were made on intuition.

Most teams skip the retrofit and ship without evals. The production system then drifts silently, and no one knows because no one is measuring.

Failure mode III — There’s no human owner waiting on the other side

We covered this in the readiness audit, but it appears here too because POCs are usually built without consulting the operator who’d receive the output. The POC ships. The output goes somewhere — a Slack channel, a Notion page, an email — and no one is responsible for acting on it.

After two weeks of unanswered Slack pings, the build is considered “not useful.” The actual problem is that the workflow had no recipient. The AI was producing real signal into a queue no one was watching.

The fix is to identify the human owner before building the POC, and to design the output around their existing workflow — not the other way around. If they read email, the output is an email. If they live in Linear, the output is a Linear ticket. Don’t make the operator change their habits to receive the AI’s output. The AI changes its habits.

Failure mode IV — The trigger was hand-fired in the demo

In the POC, someone runs the workflow by hand. They click “process,” the model runs, the output appears. The team sees it work and assumes the production version will look the same.

Production needs an automated trigger — a webhook from another system, a scheduled job, a queue listener, an API endpoint. None of those existed in the demo. Building them is engineering work that wasn’t scoped, because in the POC it didn’t exist.

We’ve watched this kill a project that had a 95%-accurate document classifier. The classifier was excellent. There was no upstream system that knew when to call it. Building that trigger took six weeks of engineering time the team hadn’t budgeted, and by the time the trigger shipped, the original sponsor had moved teams.

The day-60 ship cycle

The cycle below is what we run on every workflow build. It’s organized around the assumption that all four failure modes are present unless explicitly addressed. The discipline is sequencing: instrument first, model later.

Weeks 1–2 — Instrument first, model later

Before any model code is written, we build the eval harness skeleton and the trigger. The harness has at least 50 test cases drawn from real production data. The trigger has a no-op handler — when the trigger fires, the system records that it fired and returns immediately.

At the end of week two we have a workflow that does nothing, instrumented from trigger to eval. We can swap the no-op for a real model call without touching either side. This is what makes the next four weeks fast.

Weeks 3–4 — Ship behind a real trigger with a kill switch

The model goes in. The output goes to a shadow channel — Slack, Notion, wherever the human owner reads. The kill switch is a feature flag the operator can toggle without an engineer. If the outputs are noisy, the operator turns the flag off and the workflow stops.

The model is in production traffic, but the production user is the operator inside the team — not the end user. This is deliberate. The team learns the failure modes on a forgiving audience first.

Weeks 5–6 — Eval discipline + monitoring

Now we exercise the eval harness on every prompt change. We add regression alerts: if a deploy drops accuracy by more than five percent, the deploy is blocked. We add monitoring on the production traffic — counts, latencies, error rates, output distribution.

By the end of week six, the operator is using the system daily. The eval set has grown to include the edge cases the operator flagged. The team has confidence that a model change won’t silently break things.

Weeks 7–8 — Handoff and day-one operability

The runbook is written. The model-swap policy is documented. The eval harness is owned by someone on the team. The kill switch is wired and tested. We do a half-day enablement session with the team and walk through three scenarios: a model upgrade, a regression, and a production incident.

On day 60, we leave. The system runs. The team can extend it without us in the room. That’s what shipped means.

Why this works

The four failure modes share a common cause: in a typical POC, the model is the first thing built and everything else is retrofitted. The cycle above inverts that. The model is the last thing added. Everything that has to support the model is built first.

This is uncomfortable for teams that want to see the model working in week one. We tell them: you can have the model working in week one, or you can have the system running on day 60. Pick one. Most teams pick the wrong one and end up with neither.

What to do with this

If you have a POC that’s been “almost ready to ship” for more than three months, run it against the four failure modes. The one that’s killing you is usually obvious in 20 minutes.

If you’re about to start a POC, sequence the work so the eval harness and the trigger are in place before the model is. It feels slower. It is much faster.

The next chapter on eval harnesses and AI workflow governance goes deeper on what an eval harness actually is and what governance looks like once the system is running.