Observability Is Not Evals

Reading time ~15 min · Last reviewed Sep 2026 · Quiz at the end

LangChain's State of Agent Engineering report (2026), a survey of 1,340 practitioners run from November 18 to December 2, 2025, found a gap worth sitting with: 89% of organizations said they had implemented some form of observability for their agents, while only 52.4% said they run offline evaluations on test sets, and 37.3% run online evals. Read it with the usual caveats: it is one vendor's survey of an audience that skews toward teams already using agent tooling, and "some form of observability" is a low bar. But the ratio is the finding. The industry bought dashboards. It called that quality. Those are not the same purchase, and the gap between them is where a large share of confidently wrong production agents currently live, fully monitored, generating beautiful traces of their own mistakes.

This lesson draws the line between the two disciplines as sharply as possible, because the confusion between them is not a semantic quibble — it's a resourcing decision that determines whether your team spends its next quarter on the thing that actually predicts whether you can trust a change, or on the thing that only tells you what already happened.

The one-line distinction

Observability tells you what the agent did. Evals tell you whether it should have done it.

Say it a second way, because the distinction is easy to nod along with and then quietly forget under deadline pressure: a trace, a dashboard, a token count, a latency graph — every one of these is a record of behavior. None of them, on their own, contains a judgment about whether that behavior was correct. You can have a complete, beautifully instrumented trace of an insurance claims agent approving a duplicate payment, timestamped, with every tool call logged and every token counted, and the trace will tell you nothing is wrong. It just recorded what happened. Judging whether what happened was right requires a separate artifact — a definition of correct, applied systematically — and that artifact is an eval.

Why the gap exists

The 89/52 split isn't a random gap in engineering diligence. It reflects something true about the nature of each discipline: observability is a product you buy, and evals are a discipline you build. That difference in acquisition cost explains almost the entire gap. Observability tooling — tracing, logging, dashboards, cost and latency telemetry — is a mature, well-understood product category with vendors competing to make adoption easy. You install an SDK, you get traces. The engineering lift is real but bounded, well-documented, and largely the same regardless of what your agent actually does. It is, in the most literal sense, a checkbox you can buy your way toward. Evals have no equivalent shortcut, because an eval suite is not generic — it encodes what "correct" means for your specific workflow, and no vendor can sell you your organization's definition of a correctly adjudicated claim or a correctly prioritized service outage. Building that definition forces a question most enterprises have successfully avoided asking out loud for years: what does "good" actually mean here, specified precisely enough that a person or a program can check it? That question is uncomfortable because the honest answer is often "it depends on which senior adjuster you ask," and surfacing that disagreement is harder and slower than installing a tracing SDK. Teams under deadline pressure buy the thing that's buyable and defer the thing that requires a real specification.

Necessary, not sufficient

None of this is an argument against observability — it is a genuinely necessary layer, and skipping it is its own kind of negligence. When an eval flags a regression, or a customer escalation lands, you need traces to diagnose why: which tool call went wrong, what context the model saw, where in the reasoning chain the failure originated. Observability is the diagnostic instrument. Evals are the thing that tells you a diagnosis is needed in the first place.

The trap is treating the diagnostic instrument as if it were the diagnosis. A claims agent can run for months with immaculate traces — clean tool calls, reasonable latency, no errors thrown — while systematically misapplying a coverage exclusion on a specific policy type, because nothing in the trace says "this was wrong," only "this is what happened." The failure is invisible to observability by construction, because observability was never designed to know what correct looks like. It takes an eval — someone or something checking the output against a known-correct answer — to surface that kind of silent, well-behaved, confidently wrong pattern.

The velocity argument

The most counterintuitive claim in this lesson, and the one worth remembering when eval work gets deprioritized under a deadline, is that evals are not primarily a safety expense — they're a speed advantage. You will see this stated as a survey statistic ("eval-using teams ship several times more agents"); treat that framing with suspicion. The LangChain survey above does not show it, and its own data hints the causality may run the other way, with eval practice maturing among teams that already have agents in production. The argument does not need a survey. It is mechanical, and it should change how you think of evals if you've been treating them as the careful-but-slow option next to the fast-but-risky path of shipping on vibes. The mechanism is straightforward once you sit with it: every change to a prompt, a retrieval step, a tool, or a model version raises the same question — did this help or hurt? Without an eval suite, answering that question means a meeting: gather examples, argue about whether the new behavior is better or worse, rely on a few people's spot-checks and institutional memory, wait for someone senior enough to sign off on a gut call. That process takes days, and it doesn't scale past the second or third change of the week. With an eval suite, the same question is answered by running the suite: minutes, not meetings, and a number instead of a debate. Teams that can answer "did we break it" in minutes ship far more changes per quarter than teams that need a meeting every time, because the cost of trying something and finding out it was wrong drops by orders of magnitude. Evals are velocity infrastructure. The safety benefit is real, but the reason to fund evals before the next feature is that every later change gets cheaper to make.

Starting from zero: real cases beat synthetic ones

Teams building their first eval suite often reach for scale first — generate a thousand synthetic test cases with an LLM, cover every edge case imaginable, call it comprehensive. This produces an eval suite that looks impressive in a slide and teaches you almost nothing, because synthetic cases encode the test-writer's assumptions about what's hard, not the actual distribution of what goes wrong in your production workflow. Twenty real cases pulled from actual production history — actual claims that were actually misadjudicated, actual customer requests that actually confused the system — beat a thousand synthetic ones every time, because real cases carry the specific, unglamorous messiness (contradictory source documents, ambiguous phrasing, edge-of-policy scenarios) that synthetic generation systematically smooths over.

Start small and real: pull 20 to 50 actual cases from production or historical records, weighted toward the ones that were hard, disputed, or previously got escalated. Then get them labeled by the people who own the workflow — the senior claims adjuster, the dispatch supervisor, the CRM discount policy owner — not by the engineering team guessing at what correct looks like. This isn't a nicety; it's the only way the eval suite ends up encoding the organization's actual definition of correct rather than the engineering team's approximation of it.

Graded rubrics versus pass/fail

Not every task has a single correct answer that a pass/fail check can capture cleanly. Some do — did the tool call use the correct account ID, is the output valid JSON, does the approved discount fall within policy bounds — and those should be simple binary checks, because forcing a rubric onto a genuinely binary question just adds noise. But tasks like "summarize this claim for the adjuster" or "draft this customer response" have a quality gradient, not a single right answer, and pass/fail collapses that gradient into a false binary that either fails good-enough outputs for missing an arbitrary bar or passes mediocre ones for clearing it.

Design the rubric before you look at any model output, the same discipline a good hiring rubric requires before you look at any resume — otherwise you end up unconsciously adjusting the bar to fit whatever the model happened to produce, which defeats the point of having a rubric at all.

LLM-as-judge: useful, but calibrate it first

Using a model to grade another model's output against a rubric is an increasingly common way to scale evaluation past what human reviewers can keep up with, and it can work well — but only after you've proven it agrees with human judgment on cases where you already know the human answer. An LLM judge that hasn't been checked against real labeled cases is not a metric; it's an opinion with a decimal point attached, and decimal points make opinions look more rigorous than they are.

Calibration workflow, minimum viable version:
1. Take 30-50 cases already labeled by human domain experts (the golden
   set from the section above).
2. Run the LLM judge against the same cases, using the same rubric.
3. Compare: where judge and human agree, note it. Where they disagree,
   read the case and figure out why — ambiguous rubric wording, judge
   bias toward verbose answers, a genuine judge error.
4. Refine the rubric or the judge prompt until agreement is high enough
   to trust (teams commonly target 80-90% agreement, adjusted for how
   consequential the workflow is).
5. Re-run calibration whenever the rubric, the judged task, or the judge
   model changes. Calibration is not a one-time certificate.

Skipping calibration is how teams end up with an eval dashboard showing a reassuring 94% quality score that means nothing, because the judge was never checked against a human who actually knows what a correct claim summary looks like. An uncalibrated LLM-as-judge score is observability wearing an eval's clothing: it looks like a quality signal, traces beautifully in a dashboard, and carries none of the actual verification an eval is supposed to provide.

Traces as optimizer fuel

Everything above treats traces as the diagnostic instrument a human reaches for after an eval flags a problem. In 2026 traces picked up a second customer: the automated harness optimizers covered in lesson 22. These systems propose a change to the harness, run the eval, read what happened, and propose the next change. The eval score tells them whether a candidate won. The traces tell them why, and that is the part the next proposal depends on.

The clearest evidence comes from Meta-Harness (Lee et al., 2026). Its proposer, a coding agent, gets a filesystem holding the source code, scores, and execution traces of every earlier candidate, and browses it with grep and cat. The paper reports that a single evaluation can produce up to 10 million tokens of diagnostic information. It then ablates what the proposer is allowed to see. On its online text-classification task, a proposer given only scores reached 34.6% median accuracy; scores plus summaries of the traces reached 34.9%; full access to raw traces reached 50.0%. The authors' conclusion is that summaries "do not recover the missing signal" and may even hurt by compressing away the diagnostic detail. Best-candidate accuracy was actually lower with summaries than with scores alone.

Agentic Harness Engineering (AHE; Lin et al., 2026) takes the opposite engineering route to the same destination. Its Agent Debugger condenses roughly 10M tokens of raw traces per iteration into layered reports: an overview of cross-task root causes, per-task analyses, and claims that link back to the raw trace excerpts they came from. That last property is what separates it from the lossy summaries Meta-Harness tested. The report is an index into the raw evidence, not a replacement for it. An optimizer (or an engineer) can start at the overview and drill down to the exact tool call.

The event log as source of truth

This connects to an architectural idea covered in lesson 6: make an append-only event log the single source of truth for a session, and derive everything else from it, including the model's context, resumption after a crash, transcripts, and telemetry. Anthropic's Managed Agents (2026) keeps the session log outside the harness and outside the context window, so a crashed harness can fetch the event log and resume from the last event. DeepSeek's open harness goes further: its architecture docs (2026) state "model-visible means logged" and describe a runtime invariant checking that model requests are reconstructable from the log (lesson 23). When the log is the source of truth, observability stops being a side channel that might disagree with what actually happened. The trace is what happened, which is exactly the property an optimizer, an auditor, or an eval-case miner needs.

OpenTelemetry for agents

Traces are only fuel if they can be joined up across process boundaries. Two standards moved in 2026. The MCP 2026-07-28 specification documents OpenTelemetry trace-context propagation through request _meta keys (traceparent, tracestate, baggage), so a tool call into a remote MCP server can appear as a child span of the agent turn that made it instead of a disconnected log line on someone else's machine. The same release deprecates MCP's own Logging feature and suggests OpenTelemetry instead. Meanwhile the OpenTelemetry GenAI agent span conventions define operation names such as invoke_agent, plan, and execute_tool, with attributes like gen_ai.agent.name and token usage. They are still marked Development status, so expect churn, but naming spans consistently now makes traces portable across vendors and tools later.

invoke_agent claims-triage              gen_ai.agent.name=claims-triage
├── chat                                gen_ai.request.model=…, usage.input_tokens=18211
├── execute_tool lookup_policy          (local)
├── execute_tool check_duplicates       traceparent → MCP server span
│   └── [remote] tools/call check_duplicates
└── chat                                usage.output_tokens=412

Keep: the full span tree + raw tool inputs/outputs, keyed by trace id.
Derive: dashboards, cost reports, eval cases, optimizer reports.

The practical rule for a team starting today: instrument once, with standard span names and propagated trace context, and store raw payloads with a retention policy your governance team has approved. Then the same trace store feeds three consumers: the human debugging an incident, the eval pipeline mining hard cases (lesson 21), and whatever automated optimizer you eventually point at your harness. None of that changes the lesson's core claim. Traces still don't know what correct looks like. They are what makes the judgment, human or automated, actionable once an eval supplies it.