Reference

Glossary of Harness Engineering

A reference guide to terminology in AI agent design, deployment, and economics. These terms reflect the course thesis: an agent is a model plus its harness: the loop, context, memory, tools, execution environment, guardrails, evaluation, and economics that make it reliable and measurable in production.

135 terms · A · B · C · D · E · F · G · H · I · J · K · L · M · O · P · R · S · T · U · W · Z

A

A2A (Agent2Agent protocol): An open protocol for communication between agents from different vendors; v1.0 shipped in March 2026 and the project is now hosted by the Agentic AI Foundation. Complements MCP, which connects agents to tools and data. See lesson 16.
Action surface: The set of operations and effects an agent can execute through its tools, APIs, and integrations. Narrow action surfaces reduce blast radius and simplify guardrails; broad surfaces increase capability but raise operational risk.
Agent: A model paired with a harness: the model plus its loop, context assembly, memory, tools, execution environment, guardrails, and evaluation. The model is rented; the harness is what you build. See lesson 1.
Agent Client Protocol (ACP): A JSON-RPC protocol that standardizes how editors and IDEs drive coding agents, so one harness can serve many front ends and harnesses can spawn each other. See lesson 24.
Agent memory: Persistent state retained across agent invocations, including session history, learned user preferences, decision precedents, and intermediate computation. Distinguished from context (input for a single invocation) and organizational memory (system-wide knowledge).
Agent Skill (SKILL.md): A folder containing a SKILL.md (name, description, instructions) plus optional scripts and references, loaded on demand through progressive disclosure. Packages procedure; MCP packages connectivity. See lesson 11.
Agent team: Long-lived peer agents that coordinate through a shared task board, mailbox, or repository rather than through a single routing orchestrator. See lesson 16.
Agents Rule of Two: Meta's design rule that, within one session and without human supervision, an agent should have at most two of: processing untrusted input, access to sensitive systems or data, and the ability to change state or communicate externally. See lesson 14.
AGENTS.md: An open-format markdown file in a repository that gives coding agents project instructions; the nearest file in the directory tree applies. Works best as a short map to deeper docs, not a manual. See lesson 9 and lesson 8.
Approval fatigue: The failure mode in which reviewers asked to approve almost everything stop reading, so approvals stop meaning anything. Anthropic reported users approving 93% of Claude Code permission prompts. See lesson 13.
Audit trail: A complete chronological record of an agent's decisions, tool calls, context used, and outcomes. Required for compliance, debugging, and post-incident review; forms the basis of trace-level observability.
Autoresearch loop: An improvement loop in which a model proposes changes to a harness component, a frozen eval scores each candidate, and the loop keeps wins and reverts losses, logging everything. It speeds up hypothesis generation but does not replace the discipline of the loop itself. See lesson 21.

B

Benchmark contamination: Degradation of a public benchmark's validity as its tasks and solutions leak into model training data over time. A key reason to build private evals from your own history — a backlog of merged PRs or resolved cases is a benchmark no model has trained on, graded by checks your own team wrote (Databricks, 2026).
Benchmark rot: Benchmark tasks becoming invalid over time through dependency drift, resource mismatches, or misspecification. Terminal-Bench 2.1 repaired 28 of 89 Terminal-Bench 2 tasks. See lesson 19.
Bounded editable surface: The explicitly declared parts of a harness that an automated improvement loop is permitted to modify; everything not declared is off-limits by construction. Examples include AlphaEvolve's EVOLVE-BLOCK markers and Self-Harness's editable-surface contracts. The permission-side complement to a frozen eval.
Boundary vs judge: The distinction between the enforced limit on what an action can reach (a sandbox or VM) and whoever decides whether to cross it (a rule, a human, a reviewer agent, or a classifier). Design and test them separately. See lesson 24.
Brain/hands separation: Running the model and harness outside the sandbox and calling the sandbox as a tool, so credentials stay with the brain and sandboxes are disposable. See lesson 12.

C

Cache hit rate: The share of input tokens served from the provider's prompt cache. For agents, whose steps re-send a long prefix, it is the dominant run-cost lever and a KPI alongside cost per successful task. See lesson 6 and lesson 26.
Capability overhang: Ability a released model already has but that nobody has elicited yet. A task the model fails in plain chat may succeed once the harness gives it the right tool or structure (for example, writing and running code), so an apparent model failure is often a harness failure. See lesson 3.
Capability seam: A swappable capability designed as three parts: a service definition (interface), a provider (implementation), and a consumer (often a model-facing tool). DeepSeek Harness's unit of extensibility. See lesson 23.
Capability threshold (for self-improvement): The empirical finding (STOP; Zelikman et al. 2023) that recursive improvement loops help only above a certain base-model capability — the same loop that improved results with a strong model actively degraded them with weaker ones. Recursive structure is not a substitute for intelligence.
Capacity pricing: Pricing agent work as team capacity (pod size × seniority-blended rates × duration × escalation multipliers) rather than as a scoped project with fixed fees. Aligns incentives with utilization and quality, not output count.
Challenge card: A solution-agnostic problem statement in the form "How might we…?" used in discovery workshops to surface latent needs and divergent thinking, rather than converging on pre-selected use cases.
Classifier-based permissioning: A model decides whether to allow or escalate actions that static rules don't cover, so humans see only what it flags. Useful against approval fatigue, but it has measurable miss rates and belongs on top of deterministic controls. See lesson 13.
Code mode (programmatic tool calling): The model writes a program that calls tools as functions; the program runs in a sandbox and only its output enters context. Cuts round trips and context use, and makes a real sandbox mandatory. See lesson 11.
Compaction: Replacing part of the model-visible history with a smaller summary or trimmed version so a long trajectory can continue. Done well, it is layered: shrink or spill oversized tool results first, then summarize old history, and keep the originals in the log. See lesson 6.
Context assembly: The process of gathering, filtering, ranking, and formatting retrieval results, policies, decision graphs, and examples into a prompt that fits the context window and maximizes relevance. Core function of the harness.
Context budget: The available context window (e.g., 128K tokens) allocated across system prompt, user input, retrieval results, memory, and reasoning trace. Effective harnesses make trade-offs to prioritize high-signal information.
Context engineering: The discipline of optimizing what information, structure, and framing are presented to a model at invocation time to maximize correctness, safety, and efficiency. Encompasses retrieval, ranking, formatting, and precedent selection.
Context graph: A persistent graph encoding an organization's decisions, policies, precedents, and actors, with edges capturing relationships like justified-by, exception-to, superseded-by, and depends-on. Stores organizational judgment, not isolated facts.
Context playbook: A structured, itemized alternative to the ever-lengthening prompt: context maintained as identified bullets that are incrementally added, deduplicated, and merged by deterministic logic rather than rewritten wholesale (ACE; Zhang et al. 2025). Designed to prevent context collapse and brevity bias during iterative updates.
Context reset: Ending a session and starting a fresh one from structured handoff files instead of compacting. Some models needed resets; newer ones may not, so re-test the choice on each model generation. See lesson 15.
Context rot: Degradation of context quality over time as policies evolve, actors change roles, or decisions are reversed without updating the graph. A harness must include maintenance protocols to detect and repair rotten context.
Context window: The maximum number of tokens (input plus output) that a model can process in a single invocation. Frontier models now support 100K+ tokens; harness design must optimize within this hard limit.
Cost per successful task: Total spend, including failed attempts, divided by the number of tasks that met the bar. The headline efficiency metric for agents; per-token price is a poor proxy for it. See lesson 26.
Cross-model review: Using one model to evaluate or critique the output of another, either to catch errors the original model's priors would miss or to reduce hallucination in high-stakes tasks. Often cheaper and safer than human review for the first pass.

D

Disclosure card: A machine-readable record of the harness, model, settings, infrastructure, benchmark version, sampling, and cost behind a reported result. Without one, an agentic benchmark score is half a result. See lesson 19.
Diversity collapse: A failure mode of evolutionary and RL improvement loops in which the population degenerates into variants of one known high-reward solution. Countered with explicit mechanisms — novelty-based rejection sampling, exploration bonuses — and especially dangerous in open-ended work where the best path initially scores worse under the current evaluator.
Doom loop: An agent repeating small variations of the same failing approach. Harnesses counter it with repeat-call reminders, edit counters, and hard step or budget caps. See lesson 4.
Double Diamond: A design framework (discover, define, develop, deliver) commonly used in discovery and requirements workshops. In harness engineering, the first diamond identifies latent user needs; the second develops and hardens the agent spec.
Dual LLM pattern: A defense against prompt injection: a privileged model with tools plans using opaque references, while a quarantined model with no tools processes untrusted content. See lesson 14.

E

Egress allowlist: A default-deny outbound network policy with an explicit list of permitted destinations, usually enforced by a proxy. One of the most effective ways to break the exfiltration leg of the lethal trifecta. See lesson 12.
Eval: A test that measures whether an agent's behavior meets a specification: correctness, safety, latency, cost, or compliance criteria. Distinguished from observability (what happened) and monitoring (does it still work?).
Eval harness: The infrastructure that runs evals: test case generation, prompt templates, model execution, scoring logic, and threshold gates. Often reused across multiple agents and models.
Eval overfitting: Optimization of agent behavior to pass a narrow eval set, often at the cost of real-world performance on out-of-distribution inputs. Prevented by diverse golden sets, regression suites, and staged readiness gates.
Eval awareness: A model recognizing that an input is a test rather than real use, which can change its behavior. A reason to make eval cases look like production traffic. See lesson 18.
Evaluator agent: A separately prompted agent that grades another agent's output against criteria agreed in advance and can fail it. More reliable than self-evaluation, but it needs tuning toward skepticism. See lesson 20.
Event log: The append-only record of everything that happens in a session. The model's context is derived from it, which makes resume, fork, replay, evaluation, and audit possible. DeepSeek Harness's rule: "model-visible means logged." See lesson 6.
Evolutionary harness search: Optimizing harness or workflow designs by maintaining a population of candidates, using an LLM as the mutation operator, and selecting on eval fitness (AlphaEvolve, AFlow, Darwin Gödel Machine). Effective when candidate fitness is fast, objective, and cheap to evaluate; struggles where evaluation is slow, ambiguous, or heuristic.
Execution environment: Where an agent's actions actually run and what they can reach: host process, OS-level sandbox, container, microVM, or remote VM. A first-class harness subsystem. See lesson 12.
Execution world: A set of providers (filesystem, subprocess, terminal, language server) that must move together, so moving execution to a remote sandbox moves all of them at once. DeepSeek Harness term. See lesson 12.

F

Fail closed: When a safety mechanism can't be enforced, the action does not run, rather than running unconfined. See lesson 12.
Filesystem-as-memory: A long-horizon design pattern in which durable state — experiment logs, code diffs, error traces, past rollout trajectories — lives in files rather than the context window. Because reading, writing, and editing files is a foundational LLM skill, file-based memory automatically benefits as core model capability improves.
Fine-tuning: Retraining or adapter-tuning a model on domain-specific examples to improve task performance. In harness engineering, fine-tuning is often deferred in favor of context engineering, since strong context often outperforms weak fine-tuning at lower cost.
Forward-deployed engineer (FDE): A technical leader embedded in a business unit or customer organization to translate requirements into agent specs, shepherd rollout, and capture operational feedback for harness improvement.
Frontier model: A state-of-the-art large language model released by a leading lab, representing the current performance frontier. Harness engineering assumes deployment of frontier models, not commodity older models.

G

Goal loop: A loop in which the harness keeps the agent working toward a verifiable objective until a check passes or a budget runs out. See lesson 15.
Golden set: A curated collection of representative test cases with ground-truth labels, used to measure agent performance on high-stakes tasks and detect regressions. Kept separate from development and early-stage eval data to prevent overfitting.
Guardrail: A constraint, filter, or veto applied to an agent's behavior before or after tool execution to enforce safety, compliance, or business rules. Examples: output content filters, rate limits, escalation triggers, and action-space boundaries.
Guides and sensors: Böckeler's split of harness controls into feedforward guides that steer an agent before it acts and feedback sensors that observe after it acts; each can be computational (lint, tests) or inferential (model-based review). See lesson 9.

H

Hardened spec: A requirements specification for an agent refined through discovery, prototyping, and user feedback, explicitly capturing success metrics, failure modes, escalation paths, and autonomy boundaries. Ready for harness build-out.
Harness: Everything wrapped around a model to make it dependable and measurable in production: orchestration, context assembly, memory systems, tool integration, guardrails, evaluation, observability, and unit economics. The model is the commodity; the harness is the product.
Harness delta: Your harness's score minus a reference harness's score for the same model, tasks, and infrastructure: a defensible measure of what your harness adds. See lesson 19.
Harness engineering: The discipline of designing, building, and operating the infrastructure that makes AI agents reliable, safe, and economical. Treats the model as a component, not the solution, and focuses on context, memory, tools, guardrails, evals, and business metrics.
Harness release: A versioned, eval-gated change to prompts, tools, context policy, or defaults, shipped with staged rollout and the ability to roll back, the same way a model change would be. See lesson 21.
Human-in-the-loop (HITL): An agent design pattern in which the system flags uncertain or high-risk decisions for human review or approval before execution. Often combined with escalation thresholds and progressive autonomy.

I

Idempotency: A property of an operation: applying it multiple times produces the same result as applying it once. Idempotent agent actions simplify recovery, retry logic, and observability in distributed systems.
Indirect prompt injection: Instructions planted in content the agent reads during a task (web pages, email, tool outputs, repository files, tool descriptions) rather than typed by the user. See lesson 14.
Infrastructure noise: Score variation caused by sandbox resources, network, or infrastructure errors rather than by the agent. Anthropic measured resource limits alone moving Terminal-Bench 2 scores by about 6 points. See lesson 19.
Initializer agent: A one-time agent that sets up the environment, feature list, progress file, and first commit for the worker sessions that follow. See lesson 15.
Interleaved thinking: Model reasoning produced between tool calls and passed back in later requests. Co-designed models such as DeepSeek-V4 expect it to be preserved; harnesses that strip it remove something the model relies on. See lesson 3 and lesson 4.

J

Just-in-time context: The agent holds lightweight references (file paths, URLs, IDs) and loads content with tools when needed, instead of having everything pre-loaded into the prompt. See lesson 5.

K

Kill switch: An operator override that immediately disables an agent's action surface, halts autonomous loops, or routes all decisions to human review. A critical safety component for production harnesses.

L

Least-capability principle: The practice of granting an agent only the minimum permissions, tool access, and context required to accomplish its task. Reduces blast radius and makes guardrails simpler and more effective.
Lethal trifecta: Access to private data, exposure to untrusted content, and the ability to communicate externally, combined in one agent. Together they make prompt-injection-driven data theft possible; break at least one leg (Willison, 2025). See lesson 14.
LLM-as-judge: Using a frontier model to score the quality, safety, or correctness of another model's output in an eval. Cost-effective for subjective dimensions but can inherit the original model's biases.
Loop engineering: Designing agent loops (turn, goal, time-based, event-driven) by their trigger, stop condition, state location, and budget. See lesson 15.

M

MCP (Model Context Protocol): An open standard for connecting models to tools, data sources, and external systems. Standardizes how an agent integrates resources into its context, improving composability and reducing harness fragmentation.
Memory file: A plain file loaded into an agent's context at session start to carry knowledge across sessions, such as AGENTS.md or CLAUDE.md. Context, not enforcement. See lesson 8.
Memory poisoning: Attacker-controlled or hallucinated content written into an agent's long-term memory or knowledge base, which later returns as trusted context. See lesson 8 and lesson 14.
Meta-agent: An agent whose task is to propose, modify, or create other agents, workflows, or harnesses rather than to solve end-user tasks directly (ADAS, Meta-Harness, Hyperagents). The top rung of the optimization ladder: optimizer code improving harness code.
MicroVM: A lightweight virtual machine with its own guest kernel that boots in a fraction of a second (for example, Firecracker). Stronger isolation than a container at modest cost. See lesson 12.
Model tier: A classification of models by capability, cost, and latency (e.g., frontier, advanced, standard). A harness may route different tasks or autonomy levels to different tiers based on risk and budget.
Model–harness co-design: Designing a model and its harness contracts together: tool-call format, how reasoning is carried across turns, effort settings, and training environments. Makes models less interchangeable at the integration level even when they are interchangeable as vendors. See lesson 3.
Multi-agent system: A collection of specialized agents that collaborate or coordinate to solve problems larger than any single agent can address. Requires orchestration, message passing, and consensus protocols.

O

Observability: The ability to inspect and understand what an agent did: its inputs, reasoning steps, tool calls, context used, and outputs. Distinct from evaluation, which judges whether those actions were correct.
Orchestration: The logic that coordinates an agent's execution: sequencing tasks, routing decisions, branching based on outcomes, managing state, and handling errors. Often encoded in a state machine or workflow engine.
Orchestrator-workers pattern: A multi-agent design in which a central orchestrator decomposes tasks and coordinates specialized worker agents. Reduces hallucination and improves tractability by dividing labor.
Organizational memory: Persistent knowledge shared across an organization or harness ecosystem: policies, precedents, decision graphs, and institutional context. Distinguished from agent-specific memory and session state.

P

pass^k: The probability that all k independent attempts at a task succeed, a reliability measure from τ-bench. Contrast pass@k, where any single success counts. See lesson 19.
Per-task cost: The end-to-end cost of completing a unit of work, as opposed to per-token price. A poor correlation between the two is common: a model cheaper per token can cost more per task by working longer and reading more. Run-cost decisions require task-level measurement on your own workload, not pricing-page arithmetic.
Pilot-to-production gap: The difference between a controlled pilot environment and real-world production: volume, diversity, adversarial inputs, and edge cases often invalidate assumptions made in small-scale testing. Bridged by staged rollout and progressive autonomy.
Pod: A cross-functional team responsible for an agent or agent system in production, including engineering, product, compliance, and operations. Often treated as a capacity unit for planning and billing.
Progress file: A durable handoff note written at the end of each session so the next fresh session can resume without the transcript. See lesson 15.
Progressive autonomy: A deployment strategy in which an agent begins with high human oversight and gradually gains autonomy as it demonstrates reliability, measured by eval scores and observability data. Reduces risk and builds stakeholder confidence.
Progressive disclosure: Exposing context in layers (names and descriptions first, full instructions when relevant, bundled files only when executing) so the agent loads only what the task needs. See lesson 11.
Project pricing: Pricing agent work as a fixed-fee or scoped project with a defined endpoint. Contrasts with capacity pricing; often misaligns incentives for long-lived or evolving systems.
Projection: A pure function that turns the event log into the context the model sees. Compaction and redaction register projections instead of mutating history. See lesson 6.
Prompt ablation: Removing or changing one prompt instruction at a time and re-running evals to measure that line's effect. See lesson 10.
Prompt caching: Provider-side reuse of computation for a request prefix that exactly matches an earlier request, billed at a discount. Harnesses must keep the prefix stable, append rather than rewrite, and serialize deterministically to benefit. See lesson 6.
Prompt injection: Attacker-supplied text in a model's context that the model treats as instructions. Structural, because instructions and data share one channel, so it can't be fixed by prompting alone. See lesson 14.
Prompting inversion: The frontier-era reversal in which stronger models perform better with fewer constraints, fewer examples, and more context—invalidating older rulebook-style prompting practices. Requires harness redesign to exploit.

R

RAG (retrieval-augmented generation): A technique in which an agent retrieves relevant external documents or data before generating its response, reducing hallucination and grounding answers in sources. A core component of context assembly.
Ralph loop: A loop that restarts a fresh agent on the same prompt each iteration, with all state kept on disk. See lesson 15.
Readiness gate: An objective threshold (eval scores, regression results, escalation rates, user feedback) that must be met before an agent gains deployment, wider autonomy, or handling higher-stakes decisions. Prevents premature rollout.
Recitation: Having the agent rewrite its plan or todo file so its goals reappear near the end of the context, keeping them in attention over long runs. See lesson 6.
Recursive self-improvement (RSI): A feedback loop in which an AI uses its current capability to improve the machinery that produces its capability (Good 1965; Yudkowsky 2008). In practice the near-term path runs through improving training pipelines and deployment harnesses rather than a model directly rewriting its own weights.
Reference harness: A fixed, minimal, model-agnostic harness (such as Terminus 2) run unchanged across models so model differences can be read without harness engineering mixed in. See lesson 19.
Regression suite: A set of tests designed to detect performance degradation after model updates, prompt changes, or harness modifications. Distinct from golden-set evals; often includes edge cases and prior bugs.
Repo as system of record: Treating the versioned repository as the authoritative source of everything the agent needs to know, because what the agent can't see effectively doesn't exist for it. See lesson 9.
Retrieval precision: The fraction of retrieved documents that are relevant to the agent's task, expressed as a percentage or ranked score. High precision is critical for context assembly; low precision wastes context budget and introduces noise.
Review architecture: The design pattern for how decisions are checked in an agent system: who reviews (human, cross-model, self), what they review (outputs, reasoning, actions), and at what latency cost. Critical for safety-critical systems.
Reward hacking: An optimization loop exploiting its signal rather than achieving the intended objective: overfitting unit tests, gaming a judge model, exploiting benchmark artifacts. The reason evaluators and permission systems must sit outside any loop that evolves a harness.
Run-cost: The variable cost of executing an agent invocation: model inference tokens, tool calls, retrieval queries, and orchestration overhead. Distinct from fixed harness costs and amortized development.

S

Self-improving harness: A harness whose own code is proposed, evaluated, and merged by an automated loop — typically via weakness mining, bounded edit proposal, and no-regression validation on held-in and held-out splits (Self-Harness, Darwin Gödel Machine) — under evaluation and permission control that sit outside the loop.
Self-review trap: The failure mode in which a model or person checking its own work reproduces the same priors and blind spots that produced the error; independent review (human, cross-model, or diverse) is required to catch what self-review cannot.
Session memory: Transient state retained within a single user interaction or agent invocation, typically stored in a context window or short-term buffer. Distinct from agent memory (persistent across sessions) and organizational memory (system-wide).
Spec-driven development: A development methodology in which a hardened spec is written first, evals are derived from it, and agent implementation is guided by eval results. Reduces rework and misalignment.
Spill: Moving an oversized tool result out of the context into storage, leaving a preview plus a locator the agent can use to retrieve it. See lesson 6.
Sprint contract: Criteria for "done" that a generator and an evaluator agree on before a chunk of work starts. See lesson 15.
Step: One model request plus the tool calls it produces. Terminology varies by SDK, so always state the unit when setting limits. See lesson 4.
Stop condition: A rule that ends a turn: the model declares completion, a step or budget cap is hit, a goal check passes, or a human interrupts. Each should leave a distinct status in the log. See lesson 4.
Subagent: A child agent spawned by a parent to pursue a hypothesis, run an experiment, or handle an isolated subtask in parallel without polluting the main context. The parent acts as a small process manager — launch, inspect logs, cancel, merge — and results should persist as files and status records, not transient chat context.
System prompt: A fixed text prepended to all model invocations that defines the agent's role, constraints, output format, and reasoning approach. Acts as the primary control surface for shaping agent behavior without fine-tuning.

T

Taint tracking: Marking a session once it has read untrusted content and gating risky actions, such as network egress, on that flag. See lesson 14.
Telemetry: Structured data emitted during agent execution: latency, token usage, tool calls, escalations, and error counts. Aggregated telemetry enables monitoring, alerting, and capacity planning.
Time horizon: The length of task, measured in skilled-human time, that a model completes with a given reliability (METR). Long horizons are where harness design matters most. See lesson 15.
Token economics: The study of input/output token efficiency, context reuse, and batch processing to minimize model inference costs. A frontier-model harness must optimize token spend to stay economical at scale.
Tool: An action or function an agent can invoke: API calls, database queries, file operations, or integration endpoints. Tools expand the agent's action surface but also introduce latency, cost, and error modes.
Tool poisoning: Malicious instructions hidden in a tool's description, including descriptions that change after approval (a "rug pull"). See lesson 14.
Trace: A complete record of a single agent invocation: inputs, model outputs, tool calls, results, guardrail checks, and final action. Traces enable debugging, auditing, and observability analysis.
Turn: Everything from a user input to the point where the agent owes no further response: zero or more steps. See lesson 4.

U

Unit economics: The per-transaction or per-session cost and revenue model for an agent, typically expressed as (model + tool + orchestration cost) vs. (value captured or labor replaced). Critical for assessing viability and ROI.

W

Working state: Intermediate computation, partial results, or decision context retained during a multi-step agent task. Must be tracked in observability and often stored separately from session or organizational memory.

Z

Zombie loop: A scheduled or event-driven agent that outlives its purpose or keeps reviving unattended. See lesson 15.