Capstone: Build Your Own Harness

Reading time ~14 min · Last reviewed Sep 2026 · Quiz at the end

Twenty-six lessons of doctrine are worth nothing until you've built one real thing with them. This lesson is not an essay. It's a project brief with two halves: a working harness you build and run, and a design dossier that justifies every choice in it. Your assignment: build a harness for one real, recurring workflow from your own job. Not a demo, not a hypothetical, not a department-wide platform. One workflow, real enough that you could hand it to a colleague this quarter and they'd actually use it.

Good candidates share three properties: they recur (weekly or more often, not a one-off), they're currently done by a human doing something a model plus the right context could plausibly do, and you have — or can get — real historical examples of the work being done well. Weekly status compilation across a team's updates. Invoice triage into approve / flag / escalate. First-draft customer-reply drafting for a support queue. Field-report summarization for site visits. Pick something this concrete. If you can't name the workflow in one sentence, it's not scoped enough to start.

The deliverables

Eleven components in two parts, each with a pointer to the lesson that covers the underlying doctrine. Produce all eleven, even in rough form, before you consider the capstone done. A harness with ten strong components and one missing is not 90% complete. It's carrying an undiagnosed failure mode, exactly the kind this course has spent twenty-six lessons teaching you to find before a user does.

Part A: the working harness

Start from the lab harness in labs/minimal-harness/ in this course's repository, or from an existing harness or SDK you have chosen deliberately using the worksheet in lesson 24. The lab harness is about 250 lines of standard-library Python. It already has an explicit loop with stop conditions, an append-only event log that the context window is projected from, layered compaction, a repeat-call guard, and tool errors returned as observations. It comes with tests that run against a scripted fake model, so you can change it safely. Reading it end to end takes twenty minutes and is the fastest review of Modules 1 and 2 you will get.

cd labs/minimal-harness
python3 -m unittest -v test_harness.py      # no API key needed
python3 harness.py "your first real task"    # after setting HARNESS_BASE_URL / _API_KEY / _MODEL

A1. The loop and the log

Per lessons 4 and 6: a loop whose stop conditions you can list (explicit finish, text reply, step budget, and a cost budget you add), and an event log from which every model request can be rebuilt. Demonstrate two things: kill the process mid-task and resume from the log, and show a compaction event that kept the goal, decisions, and open work intact. If you can't replay a run from the log, you can't debug it, evaluate it, or audit it.

A2. A real execution environment

Per lesson 12: the lab's workspace path checks are not a sandbox. Move tool execution into a container or microVM with a writable workspace mount only, network egress denied by default with an explicit allowlist, resource limits, and no credentials inside. Record the policy as a file checked into your repo, not as flags someone typed once.

A3. An eval runner and a disclosure card

Per lessons 18 and 19: a script that replays your eval set (deliverable 5 below) through the harness, grades deterministically wherever it can, runs each case more than once, and writes a disclosure card: harness version, model and reasoning effort, tool list, limits, resources, pass rate with an interval, cost per successful task, and prompt-cache hit rate. Every later change to the harness is judged by re-running this script, never by eyeballing a transcript.

Part B: the design dossier

Eight documents. Each is short, specific to your workflow, and consistent with what Part A actually does. If the dossier says one thing and the harness does another, the harness is the truth and the dossier is fiction.

1. Hardened spec

Write the spec the way lesson 9 argues it should be written: policies stated as rules an adjuster or rep would recognize, worked examples showing the policy applied to real edge cases, explicit non-goals (what this agent will never attempt), and every decision point you can find resolved in writing rather than left to the model's judgment at inference time. If your workflow has an ambiguous case — a status update with no clear priority, an invoice with a discrepancy under a threshold — the spec states what happens, not "use good judgment." Put the durable parts where the agent can read them: a short AGENTS.md-style map in the repository pointing to the spec, and checks that enforce the rules mechanically wherever a rule can be checked.

2. Context inventory and assembly plan

List, for this specific workflow, what falls into each of the four context tiers from lessons 5 through 8: instructions (the spec itself), knowledge (retrieved documents, policies, historical examples), memory (what should persist across runs — prior status updates, a vendor's invoice history, a customer's past tickets), and tool outputs (live data pulled at run time). For each item, name where it actually lives today — a spreadsheet, a ticketing system, a shared drive — and how you'll get it into the agent's context without dumping the whole source system in raw.

3. Tool list with permissions and error design

Per lesson 11, list every tool the agent needs, the minimum permission each one requires (read-only where possible), and — this is the part teams skip — what the tool returns when it fails or returns an empty result. An error message that says "no records found for vendor ID 4471, verify the ID against the vendor master list" is a tool contract. A raw stack trace or a silent empty array is not.

4. Guardrail and security plan

Following lesson 13: where does a human have to be in the loop, and why there specifically rather than somewhere else in the flow? What's the kill switch — the concrete mechanism to stop the agent from taking further action if it starts misbehaving, and who has the authority to pull it? What gets logged for audit, and would that log actually answer "why did it do that" six months from now, to someone who wasn't in the room when you built it? Then apply lesson 14: list every source of untrusted content the agent reads (web pages, emails, tickets, repository files, tool outputs), every source of private data it can reach, and every channel through which it can send data out. If all three are present in one agent, which of the three legs are you breaking, and how is that enforced in code rather than in a prompt?

5. Eval set and readiness gate

Twenty real cases, pulled from actual historical work — not invented at a desk — each with a rubric describing what a correct or acceptable output looks like, per lessons 17 and 18. State your readiness gate explicitly: the specific score or pass rate this system must clear before it touches a real user, and what happens to the cases it fails today. Twenty is a floor, not a target — it's enough to catch systematic failures, not enough to catch every edge case, and you should know the difference.

6. Review architecture

Per lesson 20: who or what independently reviews the agent's output before it reaches a real user? This can be a human reviewer for the first several weeks, or a reviewer agent with fresh context and ideally a different model, but it cannot be the same process that generated the output checking itself. Name the reviewer, name the criteria it checks against, and name when (if ever) you plan to reduce the review rate as trust in the system builds.

7. Run-cost estimate and model-tier mix

Per lessons 3, 6, and 26: estimate token cost per run, state your caching assumptions and measure the actual cache hit rate from Part A, and propose a model-tier split — which slice of this workflow can run on a cheap model, which slice needs a frontier model because the failure mode (plausible-but-wrong) is expensive if it's missed. Compare the resulting per-run cost against the human baseline for the same task. If you can't state the human baseline cost, find it before you finish the capstone — it's the only number that tells you whether this is worth building at all.

8. 90-day improvement-loop plan

Per lesson 21: how will you turn the first 90 days of real usage into a better system? Name where production failures get captured, who reviews them, and how often the eval set itself gets new cases added from real misses. A harness that ships and is never revisited is a demo with better production values, not a harness.

Grading rubric

Ship it, then write it up

The capstone isn't done at deployment. Ship it to five real users — not a demo audience, five people who will actually use the output and complain honestly when it's wrong. Run the 90-day improvement loop for real: capture failures, review them, add the ones that reveal a genuine gap to your eval set, and revise the spec or guardrails when a pattern repeats. Then write up what you learned — what the spec missed that you only discovered from real usage, which guardrail placement turned out to be wrong, what the actual run-cost looked like against your estimate — publicly if you can, internally at minimum. Teaching the material, even to an audience of one skeptical colleague, is the final rep of actually mastering it. If you can't explain why your review architecture works to someone who's never read this course, you haven't finished the assignment — you've finished the build.