Benchmarking Harnesses: Disclosure, Noise, and Harness Swaps

Reading time ~16 min · Last reviewed Sep 2026 · Quiz at the end

Every launch post has a benchmark table, and every benchmark table invites the same reading: this model is better than that model by this many points. The previous lesson taught you to build an eval harness for your own workload. This one is about the numbers everyone else publishes, and about measuring the thing this course is actually about. The uncomfortable fact underneath both is that an agentic benchmark score is not a property of a model. It is a property of a model, a harness, an infrastructure configuration, a task set at a particular version, and a sampling budget, all at once. Change any one of them and the number moves, sometimes by more than the gap between the "best" and "fifth-best" model on the leaderboard.

So this lesson has two jobs. The defensive one is reading other people's numbers critically: knowing which variables were silently fixed, which were silently varied, and when a gap is inside the noise. The offensive one is measuring harnesses on purpose: running a harness A/B on your own workload with a design that can actually tell you whether the change helped. The two jobs share a toolkit, and by the end you should be able to write a disclosure card for any result you publish and refuse to act on one that lacks it.

The evidence: same model, different harness, different score

The course slogan is Agent = Model + Harness. The strongest evidence for it is a family of experiments in which the model is held fixed and only the harness changes. They are now common enough to tabulate.

LangChain (Feb 2026) took its Deep Agents harness from 52.8% to 66.5% on Terminal-Bench 2.0 with gpt-5.2-codex held constant. The changes were ordinary harness engineering: a self-verification phase enforced by a pre-completion checklist middleware, injected environment context (directory structure, available tools), a loop-detection middleware that nudges the agent after repeated edits to the same file, and a "reasoning sandwich" that spends more reasoning effort on planning and verification than on the middle of the task. The post's own summary is blunt: "We only tweaked the harness and kept the model fixed."

Automated search produces the same shape. Agentic Harness Engineering (Lin et al., 2026) evolved a coding-agent harness over ten iterations and lifted Terminal-Bench 2 pass@1 from 69.7% to 77.0%; the project README identifies the fixed model as GPT-5.4. The paper reports the evolved harness surpassing the human-designed Codex-CLI harness (71.9%) on the same setup, and its ablations attribute the gains to tools, middleware, and long-term memory rather than the system prompt. Meta-Harness (Lee et al., 2026) reports that its discovered harnesses surpass the best hand-engineered baselines on Terminal-Bench 2. How those search loops work is lesson 22's subject; here the point is only that they exist and that the model column never changed.

Cost moves as well as capability. Databricks (2026), discussed in lesson 5, ran the same model at the same thinking effort through different harnesses and saw per-task cost differ by more than 2x at unchanged quality.

The most dramatic case is also the least verified, and it is worth studying precisely for that reason. DeepSeek's V4-Pro general-availability post (Aug 2026) publishes its benchmark results as an image. That image lists 87.9 on "Terminal Bench 2.1" for DeepSeek-V4-Pro-0813, with no column saying which harness produced it. According to FindHarness's coverage, the 87.9 came from DeepSeek's own harness, and a third-party rerun by Vals AI under the Terminus 2 reference harness reported 54.68. The coverage itself calls its sources community reports rather than a DeepSeek statement. Treat the 33-point gap as reported, not established. But notice what made it possible: a headline number published as a picture with no harness disclosed. The internals of DeepSeek's harness are lesson 23's job.

Disclosure: what a result must state

The research community has started saying this out loud. The position paper "Stop Comparing LLM Agents Without Disclosing the Harness" (Zhang et al., 2026) argues that the execution harness is often a stronger determinant of agent performance than the model it wraps, that harness-induced variance can substantially exceed model-induced variance, and that this includes cases where model rankings reverse. They call for a harness-aware evaluation framework with a disclosure standard and a variance-decomposition protocol, so that infrastructure improvements stop being credited to models.

The paper frames the standard at the level of context construction, tool interaction, orchestration, and verification. For practical use you need something more concrete: a list of fields that, if any one is missing, means you cannot reproduce or compare the result. The checklist below is this course's synthesis, drawn from the variables the sources in this lesson show to matter. It is not quoted from the paper.

Put it in a machine-readable card that lives next to every result you publish or accept, in the same repository as the eval suite. The format below is a template; adapt field names to your tooling.

# disclosure-card.yaml  (one per reported result)
result:
  benchmark: terminal-bench
  benchmark_version: "2.1"          # never just "Terminal-Bench"
  tasks_run: 89
  tasks_excluded: []                # list any, with reason
  metric: pass@1                    # pass@1 | pass@k | pass^k
  runs_per_task: 5
  score: 0.642
  ci95: [0.601, 0.683]              # bootstrap over tasks
harness:
  name: our-agent
  version: "git:4f2c9ab"
  tools: [bash, edit_file, read_file, search]
  tool_descriptions_ref: "prompts/tools@4f2c9ab"
  system_prompt_ref: "prompts/system@4f2c9ab"
  max_steps: 150
  max_wall_time_s: 3600
  per_command_timeout_s: 600
  retries_on_infra_error: 1         # infra retries are not attempts
model:
  id: "<provider model id, exact>"
  reasoning_effort: high            # or per-phase map
  temperature: default
infrastructure:
  sandbox: docker
  cpu: {guaranteed: 2, limit: 6}
  memory_gb: {guaranteed: 4, limit: 12}
  network: allowlist
  infra_error_rate: 0.012
economics:
  cache_hit_rate: 0.91
  mean_cost_per_task_usd: 0.84
  cost_per_successful_task_usd: 1.31
reference:
  harness: terminus-2                # same model, reference harness
  score: 0.577
  ci95: [0.534, 0.620]

Infrastructure noise: the variable nobody reports

Even with model and harness pinned, the machine the task runs on moves the score. Anthropic (Feb 2026) ran Terminal-Bench 2.0 under six resource configurations, from strictly enforced limits to fully uncapped, and found a 6-point gap between the least and most resourced setups (p < 0.01). Infrastructure error rates fell from 5.8% under strict enforcement to 0.5% uncapped. A moderate 3x headroom cut errors to 2.1% while keeping the score lift small. On SWE-bench the effect was smaller, 1.54 points across RAM varied up to 5x, which is itself informative: sensitivity to infrastructure is a property of the benchmark, and terminal tasks that compile, install, and run things are the most sensitive.

Their recommendation is the threshold you should carry around: "Leaderboard differences below 3 percentage points deserve skepticism" until the eval configuration is documented and matched. They also recommend specifying both a guaranteed allocation and a hard kill threshold per task rather than one pinned value, treating resource configuration as a first-class experimental variable, and running evals at different times and days to average out infrastructure noise.

Two mechanisms are at work, and they are worth separating because they call for different fixes. The first is infrastructure failure: an out-of-memory kill or a network timeout that ends a run the agent would otherwise have finished. That is pure noise, and it should be counted, reported (the infra_error_rate field above), and retried separately from agent attempts. The second is subtler: more resources let certain strategies succeed at all, such as a big parallel build or a heavyweight test suite. That is not noise. It is a real capability difference, just not the one the leaderboard column claims to measure.

Benchmark rot and contamination

The task set is the other thing that silently changes. Benchmarks decay in two ways: tasks break, and tasks leak.

Tasks break

Terminal-Bench 2.1 fixed 28 of the 89 tasks in version 2.0. Nine broke because of external dependencies that change over time (internet access makes a task a moving target), eight had resource budgets too small for valid solutions to complete consistently, and eleven were misspecified, with instructions that disagreed with the tests. Nearly a third of a flagship benchmark needed repair within its first generation. The maintainers report results as "representative agent-model pairs" rather than as model scores, and most pairs improved on 2.1, which means some share of what looked like agent failure on 2.0 was benchmark failure. The practical consequence: "Terminal-Bench 2" is not a version. Any comparison across 2.0 and 2.1 numbers is invalid by default.

Tasks leak

OpenAI's post "Why we no longer evaluate SWE-bench Verified" (2026) documents both failure types at once. It audited 138 problems that o3 did not consistently solve over 64 independent runs and found that at least 59.4% of them had flawed tests that reject functionally correct solutions. It also found evidence that frontier models had seen the problems or solutions during training. Its conclusion is that gains on the benchmark no longer reflect real improvements in software engineering ability, and it recommends SWE-bench Pro (Deng et al., 2025), which it says empirically seems to suffer less from contamination. SWE-bench Pro's design choice is the transferable lesson: 1,865 long-horizon problems from 41 repositories, partitioned into a public set, a held-out set from private repositories, and a commercial set from proprietary codebases, so that a slice of the benchmark cannot have been trained on.

Note the audit's selection trick, because you can reuse it on your own golden set: look first at the tasks a strong system keeps failing. That is where broken tests and bad specifications concentrate. A task that no configuration ever passes is more likely to be a defective task than a hard one.

Reference harnesses: run one beside your own

If harness choice moves scores by double digits, you need a fixed point: a harness that everyone runs unchanged, so that model differences can be read off it without harness engineering mixed in. That is what a reference harness is.

The Terminal-Bench team built Terminus as a neutral testbed for evaluating models as terminal agents: a single tool (an interactive tmux session the model sends keystrokes to), no model-specific tuning, and support for virtually any model through LiteLLM. Terminus 2 is the version third-party leaderboards use as their common harness; both Vals AI and Artificial Analysis state that they run Terminal-Bench 2.1 with Terminus 2, the latter reporting pass@1 averaged over three repeats per task. Harbor is the framework that runs this: any agent, any model, any task set, in parallel sandboxes, with flags for concurrency and attempts.

# Same model, two harnesses, same tasks, same attempts.
# Check `harbor run --help` for your version's agent names and flags.
harbor run -d "<org/terminal-bench>" -a terminus-2   -m "<model>" -n 32 -k 5
harbor run -d "<org/terminal-bench>" -a "<your-agent>" -m "<model>" -n 32 -k 5

A minimal reference harness is deliberately not the best harness. Its value is that it is boring and fixed. Running it beside your own gives you three things you cannot get otherwise:

• A harness delta you can defend. Your harness minus the reference, same model, same tasks, same infrastructure, is the honest measure of what your harness engineering is worth. It is the number the DeepSeek coverage says was missing from the headline.
• A model-upgrade signal free of harness effects. When a new model arrives, run it under the reference first. If it gains ten points there and two under yours, your harness is compensating for weaknesses the new model no longer has, and some of it is now dead weight. Anthropic's (Mar 2026) observation that "every component in a harness encodes an assumption" about what the model cannot do on its own becomes measurable.
• An infrastructure canary. If the reference score moves between runs while nothing changed, your infrastructure moved. That is the noise floor made visible.

Metrics that survive scrutiny

Cost per successful task

Pass rate alone rewards harnesses that buy points with tokens: more steps, more retries, more reasoning. Cost per task alone rewards harnesses that give up early. The metric that captures both is total spend divided by tasks solved. Weinberger & Hozez (2026) frame coding-agent efficiency exactly this way, as cost per successful task induced by the agent trajectory, and argue that token and cache counts are measurements of that trajectory rather than targets to optimize directly. A harness that cuts tokens by trimming verification may lower cost per task and raise cost per success. Make cost per successful task the headline, and report pass rate and cost per attempt beside it so readers can see which side moved.

Variance: runs, intervals, and pass^k

Agent runs are stochastic. The same harness, model, and task can pass on Monday and fail on Tuesday. That has three consequences.

First, run each task several times. Artificial Analysis uses three repeats; five is a reasonable default for a harness A/B if you can afford it. Second, put a confidence interval on the score, bootstrapped over tasks, because tasks rather than runs are the dominant source of variance in small suites. With 89 tasks, a single pass/fail per task gives a 95% interval of roughly plus or minus ten points around a 60% score, which is why a 3-point gap from one run means nothing.

Third, choose the aggregate that matches deployment. pass@k asks whether any of k attempts succeeded. It fits settings where a verifier can pick the winner, such as a test suite that confirms the fix. pass^k, introduced by τ-bench (Yao et al., 2024), asks whether all k attempts succeeded: a reliability metric for settings where each user gets one attempt and you care whether the agent behaves consistently. τ-bench found GPT-4o succeeding on under 50% of tasks and falling below 25% pass^8 in its retail domain. pass@k climbs as k grows; pass^k falls. A harness change that raises pass@5 while lowering pass^5 made the agent more creative and less dependable, and a customer-facing product usually wants the second property.

from math import comb
import random

def pass_at_k(n, c, k):   # P(at least one of k succeeds), n runs, c passes
    return 1.0 if n - c < k else 1 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n, c, k):  # P(all k succeed)  -- tau-bench's pass^k
    return comb(c, k) / comb(n, k)

def paired_delta_ci(a, b, iters=10_000, seed=0):
    """a[i], b[i] = pass rate of harness A and B on task i (same tasks).
    Bootstrap over tasks; returns mean delta and 95% interval."""
    rng, n = random.Random(seed), len(a)
    deltas = []
    for _ in range(iters):
        idx = [rng.randrange(n) for _ in range(n)]
        deltas.append(sum(b[i] - a[i] for i in idx) / n)
    deltas.sort()
    return sum(b[i] - a[i] for i in range(n)) / n, \
           (deltas[int(.025 * iters)], deltas[int(.975 * iters)])

The bootstrap is paired: it resamples tasks and keeps each task's A and B results together. That matters because task difficulty varies enormously, and pairing removes it from the comparison. An unpaired test on the same data will report a much wider interval and miss real improvements.

Running a harness A/B on your own workload

Public benchmarks tell you about someone else's tasks. The decision you actually face is whether a harness change is better on yours. The protocol below ties together lesson 18's golden set and lesson 21's frozen eval.

1. Freeze the instrument. Pin the golden set version, the grader, the model ID, reasoning effort, and infrastructure limits. Write the disclosure card before running anything. Only the harness field may differ between arms.
2. Pre-register the decision. State the metric (cost per successful task, with pass rate as a guardrail), the minimum effect you care about, and the rule: ship only if the paired 95% interval on the delta excludes zero in your favor and no guardrail metric regresses beyond its tolerance. Writing this down first is what stops you from choosing the metric that happened to improve.
3. Run both arms together. Same tasks, same number of attempts per task, interleaved in time so both see the same infrastructure conditions. Include the reference harness as a third arm if you can.
4. Separate infrastructure errors. Retry infra failures without counting them as attempts, and report their rate per arm. A harness whose "improvement" coincides with a lower infra error rate may have just asked for less memory.
5. Read traces on flips. Look at tasks that went fail-to-pass and pass-to-fail. A real improvement has a mechanism you can see in the traces. A flip pattern with no visible mechanism is usually noise, and a gain concentrated in tasks with suspicious traces is usually reward hacking (lesson 17).
6. Check held-out tasks. If the harness change was developed while looking at the golden set, confirm on a split you did not look at. Lesson 21's held-in and held-out discipline applies to hand-tuning as much as to automated search.

Reading a leaderboard: a short checklist

When a number crosses your desk, whether in a vendor pitch, a launch post, or a colleague's slide, run it through five questions before it influences a decision:

• Which harness, at which version? If the answer is "the vendor's own," find the reference-harness number for the same model and read both.
• Which benchmark version, and were any tasks excluded?
• pass@1, pass@k, or best-of-something? How many runs, and is there an interval?
• Is the gap bigger than about 3 points, and was the infrastructure matched?
• What did a successful task cost?

Most published numbers fail at least two of these. That does not make them useless. It makes them hypotheses to test on your own workload with your own harness, which is where the decision was always going to be made.