Specs and the Repo as System of Record

Reading time ~17 min · Last reviewed Sep 2026 · Quiz at the end

For most of software history, the spec was a courtesy. A product manager wrote a document, engineers skimmed it, and the real specification emerged as code — in pull requests, in Slack clarifications, in the thousand small decisions a developer makes per hour that never made it into any doc. The code was the source of truth because writing code was the expensive, durable act, and English prose was cheap and disposable by comparison.

That relationship has inverted, and it happened quietly enough that a lot of engineering organizations haven't updated their process to reflect it. When an agent can generate a working implementation from a description in minutes, the code is no longer the expensive artifact — it's disposable, regenerable, and increasingly the thing you'd happily throw away and rebuild if the spec turns out to be wrong. The spec is now the expensive artifact, because it's the thing that took the actual thinking, and it's the thing you'll need again the next time you regenerate the implementation, retarget it to a different framework, or hand it to a different agent entirely.

This lesson covers two halves of the same shift. The first is the individual spec: what a hardened one contains and how to find its gaps. The second is where specs, and everything else an agent needs to know, should live: in the repository, as a versioned system of record that the agent can read and that machines can check.

The central asymmetry

Here is the asymmetry that this lesson is organized around, and it is worth stating plainly because it explains almost every spec-related failure this course has seen in production agent work: easy specs produce hard builds, and hardened specs produce easy builds. An easy spec — "build me a tool that flags suspicious insurance claims" — costs its author five minutes to write and imposes hours or days of rework downstream, because every decision the spec didn't make gets made ad hoc, invisibly, by whoever or whatever builds the thing. A hardened spec — one that has actually resolved what "suspicious" means, what evidence counts, what the false-positive tolerance is, what happens at the boundary cases — costs its author real time up front, and the build that follows is close to mechanical. This asymmetry is not new to agents; a experienced engineering lead has always known that a vague ticket costs more in meetings than a precise one costs in writing. What's changed is the leverage. When a human developer hits an unspecified decision point, they usually stop and ask. When an autonomous agent hits the same gap, it doesn't stop — it picks something plausible and keeps going, confidently, and the gap becomes a wrong assumption baked into working code that passes a demo and fails in week three of production.

What a hardened spec actually contains

A hardened spec is not a longer spec. Length and rigor are unrelated; a ten-page requirements document can still fail to resolve a single meaningful edge case, and a page of tight prose with the right worked examples can be airtight. What makes a spec hardened is that it contains five specific things, and a spec missing any one of them will leak decisions downstream to whoever builds it. First, behavioral examples — concrete input/output pairs that show, not tell, what the system should do. "Flag claims with inconsistent damage timelines" is a description; "Given a claim reporting a roof leak dated three days after a storm event with no storm damage reported by neighboring claims in the same ZIP within 72 hours, flag as timeline-inconsistent with confidence tier B" is a behavioral example, and it resolves ambiguity a description cannot. Second, edge cases, stated explicitly rather than left to be discovered. What happens when the claimant has no prior claims history to compare against? What happens when two data sources disagree on the storm date? A spec that only describes the happy path is a spec that hasn't been hardened. Third, acceptance criteria — a test a reasonable third party could run against the output to say pass or fail, independent of how it was built. This is the connective tissue to lesson 18's eval sets: acceptance criteria that are actually executable, not just described, are most of the way to being an eval already. Fourth, explicit non-goals — what the system is deliberately not trying to do. A claims-flagging tool that explicitly does not make approve/deny decisions, only flags for human review, needs that boundary stated as forcefully as its capabilities, because agents (and engineers) will happily generalize a capability past its intended scope if nothing stops them. Fifth, and most often skipped, decision points resolved in advance — the specific list of judgment calls the spec author identified and made, rather than leaving to whoever builds it. This is different from edge cases; edge cases are about unusual inputs, decision points are about ambiguous design choices even in the ordinary path — does "suspicious" mean flagged for human review or auto-denied, does a tie in confidence scoring default to flag or default to clear, does the system version its rules and how.

A toolkit for finding your unknowns

The hard part of hardening a spec is rarely the writing — it's knowing what you don't know. Most spec gaps are invisible to their author precisely because the author's own mental model fills them in automatically, without noticing there was a gap to fill. The following five techniques are ways of exposing those gaps before they become production incidents, and they work because each one forces a different kind of surprise.

1. Blind-spot pass

Give the model your current spec and ask it directly: "What does this spec fail to specify that a competent engineer would need to know before building this?" This works because a model reading cold, without your implicit context, will surface exactly the assumptions you're too close to see. Push it past the first answer — the first pass tends to be generic ("add more error handling"); the second and third passes, prompted with "go deeper, get specific to this domain," surface the real gaps.

2. Divergent prototypes

Generate two or three independent implementations from the same spec — different sessions, ideally different models — and diff the decisions each one made differently. Where they agree, the spec was clear enough that clarity wasn't the issue. Where they diverge — one auto-denies at high confidence, another always routes to human review, a third treats missing data as low-risk while another treats it as high-risk — every divergence is a decision point your spec left unresolved. This is one of the highest-signal techniques in the toolkit because it doesn't rely on anyone, human or model, correctly guessing what's ambiguous; the ambiguity announces itself.

3. Interview-me

Instead of writing the spec and then checking it, hand the model your rough one-line request and ask it to interview you before writing anything. A well-prompted interview pass will ask the boundary questions a rushed spec author skips — what should happen on conflicting data, who is the user when there are multiple stakeholders, what's the cost of a false positive versus a false negative. This front-loads the discovery that would otherwise happen expensively, mid-build.

4. Quiz-me-back

The inverse of interview-me: after you've written a spec, ask the model to generate a quiz about it — questions a builder would need to answer correctly to implement it right — and then try to pass your own quiz. Questions you can't answer confidently point directly at unresolved decision points. This technique is cheap, fast, and catches the specific failure mode where a spec author thinks they've been thorough because the prose reads smoothly, when smooth prose and resolved ambiguity are unrelated properties.

5. Reference-as-map

When a similar system already exists — a competitor's product, a legacy internal tool, an industry-standard workflow — give it to the model as a description of the territory rather than trying to spec the destination from scratch. "Here's how our current manual claims-review process works end to end; the new system should replicate the reviewer's actual decision logic, not a simplified version of it" gives the model something concrete to map against, and concrete maps surface gaps that abstract descriptions don't.

The spec for agent systems: policies, examples, and the eval set

Everything above generalizes across software, but agent systems have a specific version of the spec that's worth naming directly, because it's the one this course will keep returning to. For an agent, the spec is not a single document — it's three things working together: the policies (the rules and constraints, analogous to the non-goals and acceptance criteria above), the worked examples (behavioral examples, but specifically chosen to cover the decision points and edge cases that matter for this domain), and the eval set. The eval set deserves special emphasis, and lesson 18 will develop it fully, but the claim to plant here is this: the eval set is the executable part of the spec. A policy document can be misread; a worked example can be memorized instead of generalized; but an eval set — a battery of concrete cases with known-correct outcomes — is the one part of the spec that can actually be run, scored, and used to verify that a build (whether written by a human or generated by an agent) satisfies the spec's intent rather than just its wording. If you've written acceptance criteria as described above, you are most of the way to an eval set already; the remaining work is making those criteria runnable rather than just readable.

// The three-part spec for an agent system, sketched

policies:
  - "Flag claims for human review; never auto-deny."
  - "Confidence tiers: A (auto-flag), B (flag with note), C (log only, no flag)."
  - "Non-goal: this system does not determine claim payout amounts."

worked_examples:
  - input: "Roof leak claim, 3 days post-storm, no neighboring claims in 72h window"
    expected: { action: "flag", tier: "B", reason: "timeline inconsistency" }
  - input: "Water damage claim, same-day as storm, 12 neighboring claims same ZIP"
    expected: { action: "log_only", tier: "C", reason: "consistent with storm pattern" }

eval_set:
  - id: "eval-014"
    input: "Claim with missing prior-history data, otherwise clean timeline"
    expected: { action: "log_only", tier: "C" }   // resolved decision point: missing data != high risk
    # this case exists BECAUSE divergent prototypes disagreed on it

The repo as system of record

A hardened spec only helps if the agent can see it. That sounds obvious until you list where a typical team's real specification lives: a design doc in a shared drive, an architectural decision made in a chat thread, a convention everyone learned in code review, a constraint that lives only in the tech lead's head. A human new hire absorbs these over months. An agent absorbs none of them.

The clearest account of what follows from that comes from OpenAI's Codex team. In "Harness engineering: leveraging Codex in an agent-first world" (OpenAI, 2026), they describe five months building an internal product in which no code was written by hand: roughly a million lines and about 1,500 merged pull requests, driven by three engineers who later grew to seven. The post's central principle is blunt: from the agent's point of view, anything it can't access in context while running effectively doesn't exist. Knowledge in documents, chat threads, or people's heads is invisible; repository-local, versioned artifacts are all the agent can see. Their response was to keep pushing context into the repo, including the chat discussion that settled an architectural pattern, because if it isn't discoverable to the agent it is as unknown as it would be to a new hire three months later.

AGENTS.md as a map, not a manual

They first tried one large AGENTS.md and report that it failed predictably: it crowded out the task and the code, marking everything important made nothing important, it rotted into stale rules, and a single blob couldn't be checked mechanically. What worked was a short AGENTS.md, roughly 100 lines, injected into context as a table of contents, pointing into a structured docs/ directory treated as the system of record: indexed design docs with verification status, an architecture map of domains and layering, a quality document grading each domain, and execution plans with progress and decision logs checked into the repo. The agent starts from a small, stable entry point and is taught where to look next, the progressive-disclosure pattern from lesson 5. The AGENTS.md format itself is an open convention; lesson 8 covers it as agent memory.

# AGENTS.md  (illustrative skeleton; keep it about this short)

## What this repo is
Claims-triage service. Flags claims for human review; never decides payouts.
Spec and non-goals: docs/product/claims-flagging.md

## Where to look
- Architecture and layering rules: docs/ARCHITECTURE.md
- Active execution plans:          docs/plans/active/
- Decision records (why, not what): docs/decisions/
- Quality grades by domain:        docs/QUALITY.md

## How to work here
- Build: make build    Test: make test    Full check: make verify
- Run `make verify` before opening a PR; CI runs the same target.
- New work over ~1 day gets a plan in docs/plans/active/ first.

## Invariants (enforced by lint; messages explain the fix)
- Dependencies flow Types -> Config -> Repo -> Service -> Runtime.
- Parse external data at the boundary; no untyped access downstream.

## When unsure
Ask in the PR description rather than guessing. Update the doc you
wished had existed.

Notice what the skeleton does not contain: coding style minutiae, a history of incidents, or the spec itself. It says where things are and which rules are enforced elsewhere. The real content lives in files that can each be reviewed, owned, and checked.

Mechanical enforcement: rules that explain themselves

Documentation alone doesn't keep an agent-written codebase coherent, and the OpenAI post is explicit that they enforce invariants rather than micromanaging implementations. Each business domain has a fixed set of layers with validated dependency directions, enforced by custom linters and structural tests that Codex itself wrote, plus "taste invariants" such as structured logging, schema naming conventions, and file size limits. The detail worth copying: because the lints are custom, they write the error messages to inject remediation instructions into the agent's context. A failing check doesn't just say what is wrong; it tells the agent how to fix it.

# A structural lint whose message is written for the agent
def check_layering(imports, module):
    for dep in imports:
        if layer(dep) > layer(module):       # e.g. service importing ui
            fail(
                f"{module} ({layer_name(module)}) imports {dep} "
                f"({layer_name(dep)}). Dependencies must flow "
                "Types -> Config -> Repo -> Service -> Runtime -> UI. "
                "Fix: move the shared logic down into the service layer, "
                "or pass it in through a Provider. See docs/ARCHITECTURE.md#layers."
            )

The same logic runs on the docs. Linters and CI jobs check that the knowledge base is current, cross-linked, and correctly structured, and a recurring doc-gardening agent looks for documentation that no longer matches the code and opens fix-up pull requests. The team also describes a garbage-collection loop: they had spent Fridays, about 20% of the week, cleaning up agent-produced mess by hand, and replaced that with "golden principles" encoded in the repo and background agent tasks that scan for deviations and open small refactoring PRs. Their summary of the direction of travel: when documentation falls short, promote the rule into code.

Guides and sensors

Böckeler (2026) gives a vocabulary that makes this design space easier to reason about. Harness controls come in two directions. Guides are feedforward: they try to steer the agent before it acts. Sensors are feedback: they observe after the agent acts and help it self-correct. Cutting across that, each control is either computational (deterministic and fast: tests, linters, type checkers, structural analysis) or inferential (semantic analysis, AI code review, LLM-as-judge). She cites AGENTS.md as an inferential guide and custom linter messages carrying self-correction instructions as a feedback control.

Two practical conclusions fall out of the grid. First, a spec is an inferential guide, and inferential guides are requests the model may not honor; the parts of the spec that must hold should also exist as sensors, ideally computational ones, which is the same argument this lesson makes for the eval set as the executable part of the spec. Second, when you find yourself adding a paragraph to AGENTS.md after an incident, ask whether it could be a lint instead. Computational sensors are cheap, deterministic, and don't compete for context until they fire. Böckeler also groups controls by what they regulate (maintainability, architecture fitness, and behaviour), which is a useful checklist for whether your sensors cover more than code style. Reviewers as sensors are the subject of lesson 20.

Why this matters more as autonomy increases

The asymmetry between easy and hard specs gets sharper, not flatter, as you give agents more autonomy and longer task horizons. A human developer building from a vague spec accumulates small corrections through code review, standups, and Slack threads — the ambiguity gets resolved through friction, slowly, but it gets resolved. An autonomous agent running a multi-step task from a vague spec accumulates compounding wrong assumptions, because each step builds on the last one's unstated interpretation, and nothing forces a pause to check. By the time a human reviews the output, the agent hasn't made one ambiguous decision — it's made twelve, consistently, in a direction the spec author never intended and would object to if asked directly. This is precisely why the toolkit above front-loads the cost of finding decision points before the build starts, rather than relying on review to catch them after. Review catches wrong code. It catches unspecified-and-therefore-inconsistent behavior far less reliably, because inconsistent-but-plausible output is exactly what a competent reviewer, skimming for correctness rather than auditing for spec coverage, will wave through.