Capstone: Build Your Own Harness
Seventeen lessons of doctrine are worth nothing until you've built one real thing with them. This lesson is not an essay — it's a project brief. Your assignment: build a harness for one real, recurring workflow from your own job. Not a demo, not a hypothetical, not a department-wide platform. One workflow, real enough that you could hand it to a colleague this quarter and they'd actually use it.
Good candidates share three properties: they recur (weekly or more often, not a one-off), they're currently done by a human doing something a model plus the right context could plausibly do, and you have — or can get — real historical examples of the work being done well. Weekly status compilation across a team's updates. Invoice triage into approve / flag / escalate. First-draft customer-reply drafting for a support queue. Field-report summarization for site visits. Pick something this concrete. If you can't name the workflow in one sentence, it's not scoped enough to start.
Common capstone failures, in order of frequency: scope too big — "automate customer support" instead of "draft first-response replies for billing-category tickets"; synthetic eval cases invented at a desk instead of pulled from real historical work; skipping the run-cost estimate because it feels like a finance exercise instead of an engineering one; and self-review only, where the builder is also the sole judge of whether the output is good. Every one of these is a doctrine violation from earlier lessons, not a new mistake — you've already been warned about each of them.
The deliverables
Eight components, each with a pointer to the lesson that covers the underlying doctrine. Produce all eight, even in rough form, before you consider the capstone done. A harness with seven strong components and one missing is not 90% complete — it's carrying an undiagnosed failure mode, exactly the kind this course has spent seventeen lessons teaching you to find before a user does.
1. Hardened spec
Write the spec the way lesson 7 argues it should be written: policies stated as rules an adjuster or rep would recognize, worked examples showing the policy applied to real edge cases, explicit non-goals (what this agent will never attempt), and every decision point you can find resolved in writing rather than left to the model's judgment at inference time. If your workflow has an ambiguous case — a status update with no clear priority, an invoice with a discrepancy under a threshold — the spec states what happens, not "use good judgment."
2. Context inventory and assembly plan
List, for this specific workflow, what falls into each of the four context tiers from lessons 4 through 6: instructions (the spec itself), knowledge (retrieved documents, policies, historical examples), memory (what should persist across runs — prior status updates, a vendor's invoice history, a customer's past tickets), and tool outputs (live data pulled at run time). For each item, name where it actually lives today — a spreadsheet, a ticketing system, a shared drive — and how you'll get it into the agent's context without dumping the whole source system in raw.
3. Tool list with permissions and error design
Per lesson 9, list every tool the agent needs, the minimum permission each one requires (read-only where possible), and — this is the part teams skip — what the tool returns when it fails or returns an empty result. An error message that says "no records found for vendor ID 4471, verify the ID against the vendor master list" is a tool contract. A raw stack trace or a silent empty array is not.
4. Guardrail plan
Following lesson 10: where does a human have to be in the loop, and why there specifically rather than somewhere else in the flow? What's the kill switch — the concrete mechanism to stop the agent from taking further action if it starts misbehaving, and who has the authority to pull it? What gets logged for audit, and would that log actually answer "why did it do that" six months from now, to someone who wasn't in the room when you built it?
5. Eval set and readiness gate
Twenty real cases, pulled from actual historical work — not invented at a desk — each with a rubric describing what a correct or acceptable output looks like, per lessons 11 and 12. State your readiness gate explicitly: the specific score or pass rate this system must clear before it touches a real user, and what happens to the cases it fails today. Twenty is a floor, not a target — it's enough to catch systematic failures, not enough to catch every edge case, and you should know the difference.
6. Review architecture
Per lesson 13: who or what independently reviews the agent's output before it reaches a real user? This can be a human reviewer for the first several weeks, or a reviewer agent with fresh context and ideally a different model, but it cannot be the same process that generated the output checking itself. Name the reviewer, name the criteria it checks against, and name when (if ever) you plan to reduce the review rate as trust in the system builds.
7. Run-cost estimate and model-tier mix
Per lessons 3 and 16: estimate token cost per run, state your caching assumptions, and propose a model-tier split — which slice of this workflow can run on a cheap model, which slice needs a frontier model because the failure mode (plausible-but-wrong) is expensive if it's missed. Compare the resulting per-run cost against the human baseline for the same task. If you can't state the human baseline cost, find it before you finish the capstone — it's the only number that tells you whether this is worth building at all.
8. 90-day improvement-loop plan
Per lesson 14: how will you turn the first 90 days of real usage into a better system? Name where production failures get captured, who reviews them, and how often the eval set itself gets new cases added from real misses. A harness that ships and is never revisited is a demo with better production values, not a harness.
Grading rubric
| Component | Absent | Basic | Solid | Excellent |
|---|---|---|---|---|
| Spec | No written spec; behavior lives only in a prompt | Policies listed, no worked examples | Policies plus examples, most decision points resolved | Every known edge case from real history is either resolved in the spec or explicitly named as a non-goal, and a new team member could apply the policy correctly from the doc alone |
| Context inventory | No tiering; everything dumped into one prompt | Tiers named but sources vague | Tiers mapped to real sources, some pruning logic | Every context item has a named source, a refresh strategy, and a stated reason it's in the tier it's in — nothing is included "just in case" |
| Tools | No tool contracts; raw API calls with no error handling | Tools listed, permissions vague | Least-privilege permissions, basic error messages | Every tool has minimum necessary permission, and every failure mode returns an error message that tells the agent exactly what to do next, tested against real failure cases |
| Guardrails | No human checkpoint, no kill switch | A human reviews output but placement is arbitrary | HITL placed at the highest-stakes decision point, kill switch exists | HITL placement is justified by a specific cost-of-error analysis, the kill switch has been tested (not just designed), and the audit log has been proven to answer "why" on a real past case |
| Eval set | No eval set, or fewer than 20 cases, or invented cases | 20 real cases, thin rubric | 20 real cases with a clear rubric and a stated gate | 20+ real cases spanning routine and edge cases, an explicit numeric readiness gate, and a documented plan for what happens to cases that currently fail |
| Review architecture | Self-review only, or no review | A human reviews, but the same person who built the system | Independent reviewer (human or agent) with defined criteria | Independent reviewer with fresh context or a different model, explicit criteria, and a stated plan for when review intensity steps down as trust is earned |
| Run-cost estimate | No cost estimate at all | A single blended cost number, no tier mix | Cost broken out by token/caching, a tier-mix proposal | Full run-cost breakdown with tier mix justified by failure-mode risk, and an explicit comparison to the human baseline cost for the same task |
| Improvement loop | No plan beyond launch | A vague intent to "monitor and improve" | Named capture point and review cadence | A concrete 90-day plan naming who reviews failures, how often the eval set grows from real misses, and what triggers a guardrail or spec revision |
The rubric is not a scoring exercise for its own sake — it's the same discipline this course has argued for from lesson 1: the harness is the product, and a product is made of specific, inspectable components, not a vibe. Grade your own capstone against this table before anyone else does.
A realistic week-by-week pace for this capstone, if you're doing it alongside a normal job: week 1, pick the workflow and pull 20 real historical cases; week 2, write the hardened spec against those cases and build the context inventory; week 3, build the tool contracts and guardrail plan; week 4, wire up a first working version and run it against your eval set, unreviewed by anyone but you; week 5, bring in an independent reviewer and revise based on what they catch that you didn't; week 6, add the run-cost estimate and the 90-day loop plan, then ship to five real users. That is a six-week build for a properly scoped single-workflow harness — if your plan is taking longer than that, the workflow is probably too big.
Ship it, then write it up
The capstone isn't done at deployment. Ship it to five real users — not a demo audience, five people who will actually use the output and complain honestly when it's wrong. Run the 90-day improvement loop for real: capture failures, review them, add the ones that reveal a genuine gap to your eval set, and revise the spec or guardrails when a pattern repeats. Then write up what you learned — what the spec missed that you only discovered from real usage, which guardrail placement turned out to be wrong, what the actual run-cost looked like against your estimate — publicly if you can, internally at minimum. Teaching the material, even to an audience of one skeptical colleague, is the final rep of actually mastering it. If you can't explain why your review architecture works to someone who's never read this course, you haven't finished the assignment — you've finished the build.
Key takeaways
- Scope the capstone to one real, recurring workflow you can name in one sentence — not a department, not a hypothetical.
- All eight deliverables are required: hardened spec, context inventory, tool contracts, guardrail plan, eval set with readiness gate, review architecture, run-cost estimate, and a 90-day improvement-loop plan.
- The eval set must be built from real historical cases, not invented at a desk — synthetic cases are one of the most common and most damaging capstone shortcuts.
- Review architecture must be independent of the builder — self-review by the same person or process that generated the output does not satisfy the requirement.
- The run-cost estimate is not optional finance homework; it's the number that tells you, compared to the human baseline, whether the workflow was worth automating at all.
- Grade the finished harness against the rubric's four levels per component — a harness with seven excellent components and one absent one is carrying an undiagnosed failure mode, not a rounding error.
- The assignment isn't finished at deployment: ship to five real users, run the 90-day loop, and write up what you learned — teaching it is the final rep of mastering it.
Check your understanding
1. A capstone project has a hardened spec, a context inventory, a tool list with permissions, a guardrail plan, an eval set of 20 real cases, and a run-cost estimate. The builder personally checks every output before it ships and calls this "review architecture." What deliverable is actually missing?
2. A builder skips estimating run-cost and model-tier mix, reasoning that it's a finance concern separate from the engineering work. Which consequence does the lesson say this produces?
3. A capstone builder chooses "automate our entire customer support function" as their project scope. Which named capstone failure does this represent?
4. Per the rubric, what distinguishes an "Excellent" eval-set deliverable from a "Solid" one?