Eval harnesses and AI workflow governance

The question we get most often from teams running AI in production isn’t “which model should we use.” It’s “a new model just came out — should we switch?” Teams that have eval discipline answer that question in an afternoon. Teams that don’t spend two weeks deliberating, flip a coin, and find out three months later it was the wrong call.

Eval discipline is the difference. Without it, every model decision is opinion. With it, model choice becomes a routine operating decision — not a strategic one.

This chapter covers what an eval harness actually is (in concrete terms, not abstract ones), when to build one, and the eight-item governance checklist we use on every embedded engagement.

What an eval harness is

Strip away the buzzwords. An eval harness has four parts:

1. A test set

Fifty to two hundred input cases drawn from real production traffic. Not synthetic data, not happy-path examples, not the cases the demo handled well. The cases that come out of production, including the ones the team has flagged as confusing or borderline.

Test sets that are too small (under 30) miss patterns. Test sets that are too large (over 500) become expensive to run on every change, and the team stops running them. Fifty to two hundred is the working range.

2. A scoring function

Something that takes the model’s output and produces a number. Three flavors:

• Rule-based. Did the output match a regex? Was the JSON valid? Did the function call have the right arguments? Cheap, fast, deterministic. Only works when the right answer is structural.
• Model-graded. Another LLM scores the output against a rubric. Useful when the right answer is semantic (“is this response helpful and accurate?”). Always paired with a sampling check — humans grade a sample to confirm the grading model isn’t systematically wrong.
• Human-graded. A human looks at the output and scores it. Slow, expensive, but the gold standard for new test cases or contested patterns.

Most production eval harnesses use a mix. Rule-based for the structural checks, model-graded for the semantic ones, with weekly human grading on a sample to keep the grading model honest.

3. A trigger

The harness re-runs automatically on every change to the prompts, the model version, the retrieval index, or any material configuration. If a developer changes a prompt and the harness doesn’t fire, the discipline is theatrical.

For most teams, this means a CI step. The eval runs as part of the build pipeline. Failing scores block deployment.

4. A regression alert

When scores drop on a deploy, someone gets paged or notified. The threshold is team-specific — usually a 5–10 percent drop from the rolling average is the alert level. Below that and the alerts become noise; above and you miss real regressions.

The alert includes the diff: which test cases got worse, what changed in the deploy, and a link to the new outputs alongside the old ones. Without the diff, the alert is “something broke,” which sends the team into investigation mode instead of decision mode.

When to build one

The cheapest time to build an eval harness is before model selection. The next-cheapest time is before any prompt iteration past about five attempts. After that, every prompt edit is happening on intuition, the team has accumulated strong opinions about what the prompts should look like, and retrofitting a harness becomes an argument about which decisions to encode.

Once the workflow has any production traffic at all, the eval harness is no longer optional — it’s a prerequisite for any further changes. We won’t take an engagement that involves modifying a production AI workflow without an eval harness in place. The risk of regression is too high and the ability to detect it is zero without the harness.

The hard truth: most teams build the harness too late. They ship the model, watch it work for a few weeks, then scramble to add evals when something starts going wrong. By then the diagnostic question has changed from “is this regressing” to “what was the original behavior we expected,” which is much harder to answer in retrospect.

The eight-item governance checklist

Once the harness exists, the discipline has to be operated. Below is what we hand off on every engagement, with the understanding that the in-house team will own all eight items on day one.

I. The test set is representative of production traffic

Sample your test cases from real production traffic, not synthetic prompts. If your production traffic skews 70 percent category A and 30 percent category B, your test set should skew similarly. A test set that’s 50/50 will overweight the rare category and produce misleading scores.

II. The test set includes edge cases the team has flagged

When the team finds an output that looks wrong in production, that input gets added to the test set immediately. Not in the next batch, not in a quarterly refresh — same day. This is how the harness grows. A static test set ages quickly and starts missing the patterns the team is actually worried about.

III. Scoring is deterministic OR has a human sampling check

Pure rule-based scoring is fine if the answer is structural. Model-graded scoring needs weekly (or per-deploy) human grading on a 10-case sample to detect grading drift. Without the sample check, you’re trusting a black-box grader that may have its own opinions about what “good” means.

IV. Re-runs on every prompt change

Wired into CI. No manual triggering. If a developer can edit a prompt and merge without the harness firing, the discipline will erode within a quarter — the team will rationalize skipping it on small changes, and the small changes will compound into a regression no one noticed.

V. Re-runs on every model change

Same principle. Whether you’re upgrading a model version, swapping providers, or changing a parameter (top-p, temperature), the harness runs and the diff is reviewed. This is what makes the “should we switch to the new model” question answerable in an afternoon.

VI. A regression threshold is defined

Write down the number. Five percent below rolling average? Ten? Different per-category? The number doesn’t need to be perfect — it needs to exist, be agreed, and be enforced. Without a written threshold, every regression alert turns into a debate about whether it’s really a regression.

VII. Output diffs are inspected before deploy

Not just scores — actual outputs. The score might hold steady while the model starts producing subtly different responses that the rubric doesn’t catch. Reviewing diffs (old output vs new output for the same input) on a sample of test cases catches the slow drift that score-only review misses.

We typically review diffs on 10–20 cases per deploy. The review takes 15 minutes. It catches the things scores don’t.

VIII. The test set evolves with production

Quarterly review. New patterns from production get added. Outdated test cases (workflows that no longer matter) get retired. The test set should always reflect the workflow as it is, not the workflow as it was when the harness was first built.

Without this maintenance, the harness goes stale within a year and the scores stop correlating with actual production quality.

Why this gives you model-swap freedom

The end-state of having all eight items in place is operational freedom on the model layer. New model ships? Run the harness. Better scores? Switch. Worse scores? Don’t. Same scores but cheaper? Switch. The decision is mechanical, takes hours, and produces a defensible answer.

Compare to teams without this discipline. New model ships, team debates for two weeks, decision is made on vibes, engineering team spends a sprint on the migration, and three months later the team realizes the new model is regressing on a category no one was tracking. The migration has to be reversed. The reversal takes another sprint. Total cost: two sprints plus three months of degraded outputs that no one could see.

The eval harness pays for itself the first time you swap a model. After that, it’s pure margin.

What “model-agnostic” actually means

Teams say they’re “model-agnostic in principle” all the time. In practice it means very little — usually that they have one model wired throughout the codebase but could theoretically swap if they had to. That isn’t agnosticism. It’s lock-in with optimism.

Real model-agnosticism has three pieces:

• The model call is abstracted behind a thin interface, so the prompt and the model are configurable separately.
• The eval harness exists and runs on every model change.
• The team has actually swapped models at least once in the last six months, even if just for a test. The capability atrophies if it’s never exercised.

Without all three, “model-agnostic” is aspiration, not architecture.

What to do with this

If you have AI in production without an eval harness, this is the next thing to build. It’s typically a one-engineer, two-week project — much smaller than people expect, because most of the work is wiring existing pieces (test set, scoring, CI) rather than inventing new infrastructure.

If you’re about to ship AI to production for the first time, the harness goes in before the model. The chapter on why most AI POCs don’t ship sequences the work week-by-week.

If you’re trying to figure out who owns this discipline in-house, the chapter on build vs. hire vs. embed covers the staffing question.