Building Eval Harnesses
Lesson 11 made the case that evals are the discipline enterprises skip because observability is easier to buy. This lesson is about what to actually build once you've accepted that argument. The single most important mindset shift is this: an eval harness is product code. It gets a repository, an owner, a review process, and a place in CI, exactly like the agent it evaluates. Teams that treat their eval suite as a side project — a folder of ad hoc scripts someone runs manually before a big demo — end up with an eval suite that rots at the same rate as any unowned code, and rot in your measurement system is worse than rot anywhere else, because it silently corrupts every decision downstream of it.
The practical question is what to build, in what order, and how to keep it honest as the system it evaluates keeps changing. This lesson lays out a structure — the eval pyramid — that scales from a team's first afternoon of eval work to a mature program running dozens of gated deployments a month.
The eval pyramid
Not every check belongs at the same layer, because not every check has the same cost or needs to run at the same frequency. Structuring evaluation as a pyramid — cheap and frequent at the base, expensive and occasional at the top — gets you fast feedback on the common cases and deep scrutiny on the ones that matter without paying deep-scrutiny cost on every single change.
| Layer | What it checks | Cost | When it runs |
|---|---|---|---|
| Deterministic checks | Schema validity, tool-call contracts, policy regexes, forbidden actions, output format | Very low — milliseconds, no model calls needed to grade | Every commit, every run, always |
| Scenario evals | Golden-set cases graded against rubrics — does the agent handle this real scenario correctly | Moderate — requires running the agent and grading output, sometimes with LLM-as-judge | On every meaningful change: prompt, retrieval, tool, or model version |
| Human review sampling | Nuanced judgment, edge cases, drift detection that automated checks miss | High — consumes expert time | On a fixed schedule (e.g., weekly sample) and always after an incident |
The base layer is where most teams under-invest relative to its cost-to-value ratio. A deterministic check that verifies every tool call matches its declared schema, that a discount value never exceeds a hard ceiling, that a claim summary never fabricates a policy number not present in source documents — none of this requires a model to grade, none of it requires a human, and all of it can run in the time it takes to run a unit test suite. It catches an entire category of failure — the ones with a clear, mechanical definition of wrong — before a scenario eval or a human reviewer ever needs to look at the case.
Push as much checking as possible down to the deterministic layer. Every failure mode you can express as a schema constraint, a regex, or a hard numeric bound is a failure mode you no longer need a human or an LLM judge to catch — and it's a check that runs on every single commit for free, which means it catches regressions the same day they're introduced instead of the same quarter.
Golden sets: building, versioning, refreshing
The scenario-eval layer runs on a golden set — a curated collection of real cases with known-correct (or expert-graded) answers, the artifact lesson 11 argued should start from 20 to 50 real production cases rather than synthetic generation. Building it is only step one; a golden set is a living artifact that needs the same lifecycle discipline as any other piece of infrastructure it sits alongside.
- Build — pull real cases weighted toward the hard, disputed, and previously escalated ones; get them labeled by the people who own the workflow, not by the engineering team's best guess.
- Version — store the golden set under version control, same as code, with a changelog. When a case is added, removed, or its label changes, that's a reviewable diff, not a silent edit. Tag every eval run against the specific golden-set version it used, so a score from March and a score from June are comparable only if you know whether the underlying set changed between them.
- Refresh — add new cases as new failure patterns emerge in production, and periodically review whether old cases still reflect current policy (a claims exclusion rule that changed last quarter makes any golden-set case built against the old rule actively misleading).
A golden set nobody refreshes stops being a test of the system and starts being a test of the system's ability to memorize its way around the test. This isn't necessarily deliberate gaming — it happens naturally as engineers iterate against the same fixed set of cases over months, tuning prompts and retrieval until performance on those specific cases converges near-perfectly, without that improvement generalizing to the broader distribution of real traffic. If your golden-set score has been climbing steadily for two quarters while production incidents haven't dropped at all, suspect this pattern first.
Case study: mining your own history for a golden set
The lesson 11 argument — start from real production cases, not synthetic generation — turns out to hold at a scale most teams never reach, and it holds for a reason worth internalizing. When Databricks (2026) set out to benchmark coding agents against their own multi-million-line, ten-plus-language codebase, they deliberately rejected the public benchmarks everyone reaches for first — SWE-bench, TerminalBench, and the rest. Two reasons, and both generalize far beyond coding agents. First, contamination: public benchmarks leak into training data over time, so a rising score can measure memorization rather than capability, and you have no way to tell which. Second, unrepresentativeness: a public suite is a test of someone else's stack, not yours, and an agent that aces it can still faceplant on your frameworks, your conventions, and your idioms. This is exactly the real-cases-over-synthetic case the previous lesson made, validated at enterprise scale — the most trustworthy golden set is one drawn from your own production reality, because nothing else is both representative and contamination-proof.
Their source was hiding in plain sight: thousands of merged pull requests a day are already a labeled dataset. The construction methodology is the part worth copying. They filtered hard before anything else — recency (so tasks reflect current frameworks and conventions rather than a codebase archaeology exercise), human-written changes only (bot commits, service accounts, and fully AI-generated changes excluded, because you want to measure the agent against human judgment, not against another machine's output), a high-quality test suite attached, self-contained scope touching few modules, and a distribution of tasks representative of the full stack rather than clustered wherever mining happened to be easiest. Then they rewrote each PR's description into a well-specified prompt that states the problem, the goal, and the constraints — while removing the solution. That last move is the subtle one: if a bug-fix PR explains why the fix is correct, leaving that explanation in hands the agent the answer and turns a hard task into a trivial one. The test files were split out and held back entirely; the build system determines which test targets depend on the touched files, and all of those run in full as the grader.
Every candidate task was hand-reviewed, despite the generation being scripted and AI-assisted — automation proposes, humans dispose. Some of the original tests had to be manually rewritten, and pointedly without AI assistance: a test that graded on exact string match, for instance, was rewritten to grade behavior instead, because exact-match grading is simply wrong for non-deterministic output and would have failed correct answers. This is the same freeze-and-review discipline this lesson keeps returning to, applied at the level of individual grading logic.
Grading was deterministic held-out test execution, full stop. They explicitly declined to use an LLM judge for correctness, with a line worth pinning above your desk: an LLM judge "rewards sounding right over being right." For a correctness benchmark — as opposed to a fluency or tone assessment — that is disqualifying, and it echoes this lesson's own bias toward pushing checks down to the deterministic layer wherever a mechanical definition of correct exists. A held-out test suite the team already wrote is the purest form of that deterministic layer.
The eval harness itself can contain an exploitable artifact, and the tell is a score that looks too good to be true. Early on, some model scores did exactly that. Manual inspection of the agent traces — observability serving the evals — found the cause: every task originated from a merged commit, and the "correct" implementation was still sitting in the git history of the worktree the agent ran in. Nothing stopped an agent with a shell from walking that history to recover the answer. The fix was to seal git history, cutting the working copy off from the repository entirely for the length of each run. The general rule outlives the specific bug: when an agent's score jumps, audit the traces before you celebrate. Benchmark artifacts are reward hacking's cheapest food — an agent optimizing a metric will find the recoverable answer, the leaked test, or the history you forgot to seal long before it does the hard work you meant to measure.
The punchline is the part to carry into your own program: any team with a backlog of merged PRs is sitting on a benchmark no model has trained on, graded by tests the team itself wrote. The raw material for a contamination-proof, representative golden set is almost certainly already in your version-control history — the work is the filtering, the rewriting-to-remove-the-solution, the holding-out of tests, and the hand review, not the sourcing.
Regression suites named after real failures
Every production incident should leave behind a permanent artifact: a regression test, named after the failure it captures, added to the suite so that specific failure can never silently reoccur. This is the same discipline mature software engineering teams already apply to bugs — a bug fixed without a regression test is a bug that will eventually come back — applied to an agent's behavioral failures instead of a traditional code defect.
claims-duplicate-invoice-2026-03:
discovered: 2026-03-14
description: >
Agent approved a second payment on an invoice already paid 11 days
earlier because the duplicate-check tool only searched the current
policy year, missing a cross-year duplicate.
input: { claim_id: "CLM-88213", invoice_ref: "INV-40021" }
expected_behavior: >
Agent must flag as a potential duplicate and escalate to human
review before issuing any payment, regardless of policy-year
boundary.
regression_check: deterministic # duplicate-check tool now searches
# across all policy years; verified
# by contract test, not model judgment
status: fixed, monitoredNaming matters more than it looks like it should. "claims-duplicate-invoice-2026-03" tells anyone reading the suite exactly what happened, when, and in what domain, without opening the file. A regression suite made of cases named test_47 through test_112 has the same nominal coverage and none of the institutional memory — six months later nobody can tell you what test_47 actually protects against, which means nobody notices when a change quietly breaks its assumptions.
Pull the last three production incidents your team handled for any agent system — the postmortems, the Slack threads, whatever record exists. For each one, check whether a regression test exists that would catch that exact failure today. If not, write one this week, named after the incident the way the example above is named. This is the highest-leverage eval work available to a team that already has an incident history and no corresponding regression suite: you are not guessing at what might go wrong, you are encoding what you already know went wrong.
Readiness gates
The eval pyramid earns its keep operationally through readiness gates: a dashboard pattern where a deployment, or an advance to a higher autonomy level (lesson 10's progressive-autonomy ladder), requires every gate to show green. No single green metric authorizes a launch on its own, because any single metric can be gamed or can miss a failure mode outside its scope — the gates work as a set.
| Gate | Example threshold | What it protects against |
|---|---|---|
| Golden-set score | ≥ 92% on the current-version golden set | Known scenario regressions across the curated case set |
| Regression failures | Zero failing regression tests | Reoccurrence of any previously fixed, named production incident |
| Production-shadow runs | N consecutive clean runs in shadow mode (agent runs alongside the live system without acting) | Failure modes the golden set didn't anticipate, surfaced against real live traffic before the agent gets write access |
| HITL escalation rate | Below X% of runs requiring human escalation | An agent that's technically passing evals but practically unusable without constant human intervention |
Tying autonomy levels to these gates, as lesson 10 described, turns "should we trust this agent with more authority" into a question with a documented, falsifiable answer instead of a confidence-based judgment call made under launch-review pressure. It also gives engineers a concrete target to build toward: not "make the agent better" in the abstract, but "get the production-shadow gate green," which is buildable, testable, and unambiguous about when it's done.
A readiness gate is only as trustworthy as the measurement behind it. A golden-set score gate is worthless if the golden set has quietly become stale or overfit; an escalation-rate gate is worthless if the escalation criteria changed last month without anyone updating the threshold. Gates enforce discipline on the system being measured — they don't enforce discipline on themselves. That's your job, and it's the subject of the next section.
Freeze the measurement before optimizing
This is the rule that disciplined eval programs follow and undisciplined ones violate constantly, usually without noticing: never change the eval and the system under test in the same cycle. If you adjust the rubric, add cases to the golden set, or change the LLM-judge prompt at the same time you're tuning the agent's prompt or retrieval logic, you have made it impossible to know whether a score change reflects the agent getting better or the measurement getting different. You have learned nothing, and you may walk away believing you learned something, which is worse than learning nothing.
The discipline in practice: pick your golden set, your rubric, and your judge configuration for a given evaluation cycle, and hold all three fixed while you iterate on the system. Make your changes, measure against the frozen eval, record the result. Only after that cycle closes — and ideally only when you have a specific reason, like a documented gap the current eval doesn't cover — do you revise the eval itself, as a separate, deliberate, reviewed change, distinct from any system tuning happening in parallel.
The most common way this discipline breaks down is subtle: an engineer tuning a prompt notices the golden set has a case that seems mislabeled, "fixes" the label on the spot, and continues tuning against the now-changed set in the same sitting. Every intention here is good, and the outcome is still a corrupted comparison — the before-score and after-score are no longer measuring the same thing. Route eval changes through the same review process as the golden-set versioning practice above, on their own timeline, never bundled with the system change they were noticed alongside.
A minimal eval case format
You don't need elaborate tooling to start. A minimal eval case needs four fields: the input, what correct behavior looks like, how to grade it, and tags for organizing and filtering the suite as it grows.
id: crm-discount-tier-boundary-001
tags: [crm, discount-policy, boundary-condition]
input:
customer_tier: "gold"
requested_discount_pct: 18
order_value_usd: 42000
expected_behavior: >
Gold-tier customers are capped at 15% without VP approval. Agent should
either cap the proposal at 15% and note the reduction, or route to
escalate_discount_approval for anything above 15% — it must not
silently approve 18%.
rubric:
type: pass_fail
check: >
Output either (a) proposes <= 15% with a stated reason, or
(b) calls escalate_discount_approval. Any other outcome fails.
grading: deterministic # policy ceiling is a hard rule, checked by code
# against the returned discount value and tool
# calls made, no model judgment required
owner: crm-discounting-team
added: 2026-05-02
source: production-case # vs. syntheticNotice the format doesn't distinguish structurally between a deterministic check and a scenario eval — both fit the same four-field shape, with the grading field indicating whether a program or a rubric-driven judge (human or calibrated LLM) evaluates the result. That consistency matters in practice: it means your entire eval suite, from the cheapest schema check to the most nuanced graded rubric, lives in one format, one repository, and one CI pipeline, instead of splintering into incompatible tooling per layer.
Running it in CI
Once cases exist in a consistent format, wiring them into CI is mechanical: deterministic checks run on every commit and block merges on failure, the same as a unit test suite. Scenario evals run on every meaningful change to prompts, retrieval, tools, or model version — this is usually not every commit, since scenario evals cost more (model calls, sometimes LLM-judge calls), but it should be automatic and required before a change reaches production, not a manual step someone remembers to run before a release. Human review sampling runs on a fixed schedule regardless of whether anything changed, because drift can happen with no code change at all — a shift in the real-world distribution of incoming cases, a change in an upstream data source's format, a model provider's silent update to a model version pinned by name but not by weights.
The organizations that get real value out of this pyramid are the ones that resist treating it as a one-time setup project. The suite is never finished — it grows every time production teaches you something the current suite didn't already know, and shrinks (deliberately, reviewably) when a case stops reflecting current policy. That is what "eval harness as product code" means in practice: it has a backlog, it has technical debt, and it has an owner whose job includes keeping it honest.
This week, write your first five eval cases in the format above for one agent workflow: two deterministic checks (schema or policy-bound violations that should never happen), two scenario cases pulled from real production history with a graded or pass/fail rubric, and one regression case named after a real past incident if one exists. Put all five in version control in a single file, and wire the two deterministic ones into your CI pipeline before you do anything else — that's the cheapest, fastest-payoff piece of this entire lesson, and it's the piece most teams skip in favor of something that feels more sophisticated.
Key takeaways
- Treat the eval harness as product code: owned, versioned, reviewed, and run in CI — not a folder of scripts someone runs before a demo.
- The eval pyramid has three layers: cheap deterministic checks (run always), scenario evals against golden sets (run on change), and human review sampling (run on schedule and after incidents).
- Golden sets need a full lifecycle — build from real cases, version like code, and refresh deliberately — or they become a benchmark the system quietly overfits to.
- Every production incident should produce a permanent, descriptively named regression test, so a known failure can never silently reoccur.
- A backlog of merged PRs is a ready-made, contamination-proof golden set (Databricks, 2026): filter and rewrite tasks to remove the solution, hold out the team's own tests as a deterministic grader, hand-review every sample, and audit traces whenever a score looks too good — benchmark artifacts like recoverable git history are reward hacking's cheapest food.
- Readiness gates (golden-set score, zero regressions, clean shadow runs, escalation rate) work as a set and tie directly to autonomy-level advancement.
- Freeze the measurement before optimizing: never change the eval and the system under test in the same cycle, or you learn nothing from the comparison.
- The eval set is the executable part of the spec — it is the one artifact that states, in a form that actually runs, what your organization means by "correct."
Check your understanding
1. In the eval pyramid, why should as many checks as possible be pushed down to the deterministic layer?
2. What is the risk of a golden set that engineers keep tuning against for months without refreshing it, per the lesson?
3. Why does the lesson recommend naming regression tests after the real incident that caused them, like "claims-duplicate-invoice-2026-03," instead of generic names like "test_47"?
4. A team notices what looks like a mislabeled case in the golden set while tuning a prompt, fixes the label immediately, and keeps iterating against the updated set in the same session. What does this lesson say about that sequence?