Observability Is Not Evals

Reading time ~12 min · Quiz at the end

Industry surveys on agent engineering practice report a gap worth sitting with: roughly 89% of teams building production agents report having observability in place, while only around 52% report having evals. Those numbers are reported, not universal facts — survey methodology on a fast-moving practice always deserves a grain of salt — but the ratio is the finding, and it shows up consistently enough across different surveys to be a real signal about where the industry's attention went. The industry bought dashboards. It called that quality. Those are not the same purchase, and the gap between them is where a large share of confidently wrong production agents currently live, fully monitored, generating beautiful traces of their own mistakes.

This lesson draws the line between the two disciplines as sharply as possible, because the confusion between them is not a semantic quibble — it's a resourcing decision that determines whether your team spends its next quarter on the thing that actually predicts whether you can trust a change, or on the thing that only tells you what already happened.

The one-line distinction

Observability tells you what the agent did. Evals tell you whether it should have done it.

Say it a second way, because the distinction is easy to nod along with and then quietly forget under deadline pressure: a trace, a dashboard, a token count, a latency graph — every one of these is a record of behavior. None of them, on their own, contains a judgment about whether that behavior was correct. You can have a complete, beautifully instrumented trace of an insurance claims agent approving a duplicate payment, timestamped, with every tool call logged and every token counted, and the trace will tell you nothing is wrong. It just recorded what happened. Judging whether what happened was right requires a separate artifact — a definition of correct, applied systematically — and that artifact is an eval.

Why the gap exists

The 89/52 split isn't a random gap in engineering diligence. It reflects something true about the nature of each discipline: observability is a product you buy, and evals are a discipline you build. That difference in acquisition cost explains almost the entire gap. Observability tooling — tracing, logging, dashboards, cost and latency telemetry — is a mature, well-understood product category with vendors competing to make adoption easy. You install an SDK, you get traces. The engineering lift is real but bounded, well-documented, and largely the same regardless of what your agent actually does. It is, in the most literal sense, a checkbox you can buy your way toward. Evals have no equivalent shortcut, because an eval suite is not generic — it encodes what "correct" means for your specific workflow, and no vendor can sell you your organization's definition of a correctly adjudicated claim or a correctly prioritized service outage. Building that definition forces a question most enterprises have successfully avoided asking out loud for years: what does "good" actually mean here, specified precisely enough that a person or a program can check it? That question is uncomfortable because the honest answer is often "it depends on which senior adjuster you ask," and surfacing that disagreement is harder and slower than installing a tracing SDK. Teams under deadline pressure buy the thing that's buyable and defer the thing that requires a real specification.

Necessary, not sufficient

None of this is an argument against observability — it is a genuinely necessary layer, and skipping it is its own kind of negligence. When an eval flags a regression, or a customer escalation lands, you need traces to diagnose why: which tool call went wrong, what context the model saw, where in the reasoning chain the failure originated. Observability is the diagnostic instrument. Evals are the thing that tells you a diagnosis is needed in the first place.

The trap is treating the diagnostic instrument as if it were the diagnosis. A claims agent can run for months with immaculate traces — clean tool calls, reasonable latency, no errors thrown — while systematically misapplying a coverage exclusion on a specific policy type, because nothing in the trace says "this was wrong," only "this is what happened." The failure is invisible to observability by construction, because observability was never designed to know what correct looks like. It takes an eval — someone or something checking the output against a known-correct answer — to surface that kind of silent, well-behaved, confidently wrong pattern.

The velocity argument

The most counterintuitive claim in this lesson, and the one worth remembering when eval work gets deprioritized under a deadline, is that evals are not primarily a safety expense — they're a speed advantage. Industry surveys report that teams using evals ship several times more agent systems to production than teams that don't. That number should surprise you if you've been thinking of evals as the careful-but-slow option next to the fast-but-risky path of shipping on vibes. The mechanism is straightforward once you sit with it: every change to a prompt, a retrieval step, a tool, or a model version raises the same question — did this help or hurt? Without an eval suite, answering that question means a meeting: gather examples, argue about whether the new behavior is better or worse, rely on a few people's spot-checks and institutional memory, wait for someone senior enough to sign off on a gut call. That process takes days, and it doesn't scale past the second or third change of the week. With an eval suite, the same question is answered by running the suite: minutes, not meetings, and a number instead of a debate. Teams that can answer "did we break it" in minutes ship far more changes per quarter than teams that need a meeting every time, because the cost of trying something and finding out it was wrong drops by orders of magnitude. Evals are velocity infrastructure. The safety benefit is real but almost incidental to the speed benefit in terms of why the surveyed teams are actually shipping more.

Starting from zero: real cases beat synthetic ones

Teams building their first eval suite often reach for scale first — generate a thousand synthetic test cases with an LLM, cover every edge case imaginable, call it comprehensive. This produces an eval suite that looks impressive in a slide and teaches you almost nothing, because synthetic cases encode the test-writer's assumptions about what's hard, not the actual distribution of what goes wrong in your production workflow. Twenty real cases pulled from actual production history — actual claims that were actually misadjudicated, actual customer requests that actually confused the system — beat a thousand synthetic ones every time, because real cases carry the specific, unglamorous messiness (contradictory source documents, ambiguous phrasing, edge-of-policy scenarios) that synthetic generation systematically smooths over.

Start small and real: pull 20 to 50 actual cases from production or historical records, weighted toward the ones that were hard, disputed, or previously got escalated. Then get them labeled by the people who own the workflow — the senior claims adjuster, the dispatch supervisor, the CRM discount policy owner — not by the engineering team guessing at what correct looks like. This isn't a nicety; it's the only way the eval suite ends up encoding the organization's actual definition of correct rather than the engineering team's approximation of it.

Graded rubrics versus pass/fail

Not every task has a single correct answer that a pass/fail check can capture cleanly. Some do — did the tool call use the correct account ID, is the output valid JSON, does the approved discount fall within policy bounds — and those should be simple binary checks, because forcing a rubric onto a genuinely binary question just adds noise. But tasks like "summarize this claim for the adjuster" or "draft this customer response" have a quality gradient, not a single right answer, and pass/fail collapses that gradient into a false binary that either fails good-enough outputs for missing an arbitrary bar or passes mediocre ones for clearing it.

Design the rubric before you look at any model output, the same discipline a good hiring rubric requires before you look at any resume — otherwise you end up unconsciously adjusting the bar to fit whatever the model happened to produce, which defeats the point of having a rubric at all.

LLM-as-judge: useful, but calibrate it first

Using a model to grade another model's output against a rubric is an increasingly common way to scale evaluation past what human reviewers can keep up with, and it can work well — but only after you've proven it agrees with human judgment on cases where you already know the human answer. An LLM judge that hasn't been checked against real labeled cases is not a metric; it's an opinion with a decimal point attached, and decimal points make opinions look more rigorous than they are.

Calibration workflow, minimum viable version:
1. Take 30-50 cases already labeled by human domain experts (the golden
   set from the section above).
2. Run the LLM judge against the same cases, using the same rubric.
3. Compare: where judge and human agree, note it. Where they disagree,
   read the case and figure out why — ambiguous rubric wording, judge
   bias toward verbose answers, a genuine judge error.
4. Refine the rubric or the judge prompt until agreement is high enough
   to trust (teams commonly target 80-90% agreement, adjusted for how
   consequential the workflow is).
5. Re-run calibration whenever the rubric, the judged task, or the judge
   model changes. Calibration is not a one-time certificate.

Skipping calibration is how teams end up with an eval dashboard showing a reassuring 94% quality score that means nothing, because the judge was never checked against a human who actually knows what a correct claim summary looks like. An uncalibrated LLM-as-judge score is observability wearing an eval's clothing: it looks like a quality signal, traces beautifully in a dashboard, and carries none of the actual verification an eval is supposed to provide.