Reference

Glossary of Harness Engineering

A reference guide to terminology in AI agent design, deployment, and economics. These terms reflect the course thesis: an agent is the sum of a model plus its harness—the orchestration, context, memory, tools, guardrails, evaluation, and business logic that make it reliable and measurable in production.

A

Action surface: The set of operations and effects an agent can execute through its tools, APIs, and integrations. Narrow action surfaces reduce blast radius and simplify guardrails; broad surfaces increase capability but raise operational risk.
Agent: A model paired with a harness: the combination of a frontier LLM, its orchestration layer, context assembly, memory systems, tool integrations, guardrails, and evaluation infrastructure. The model is the commodity; the harness is the product.
Agent memory: Persistent state retained across agent invocations, including session history, learned user preferences, decision precedents, and intermediate computation. Distinguished from context (input for a single invocation) and organizational memory (system-wide knowledge).
Audit trail: A complete chronological record of an agent's decisions, tool calls, context used, and outcomes. Required for compliance, debugging, and post-incident review; forms the basis of trace-level observability.
Autoresearch loop: An agent pattern in which the system autonomously retrieves sources, cross-references claims, identifies gaps, and re-queries to build confidence in its answer. Reduces hallucination risk by making reasoning verifiable.
Capability overhang: The gap between a model's theoretical ability and what deployed systems actually extract from it, often due to weak prompting, context waste, or guardrails that suppress capability. Reducing overhang is a core harness engineering concern.

B

Benchmark contamination: Degradation of a public benchmark's validity as its tasks and solutions leak into model training data over time. A key reason to build private evals from your own history — a backlog of merged PRs or resolved cases is a benchmark no model has trained on, graded by checks your own team wrote (Databricks, 2026).
Bounded editable surface: The explicitly declared parts of a harness that an automated improvement loop is permitted to modify; everything not declared is off-limits by construction. Examples include AlphaEvolve's EVOLVE-BLOCK markers and Self-Harness's editable-surface contracts. The permission-side complement to a frozen eval.

C

Capability threshold (for self-improvement): The empirical finding (STOP; Zelikman et al. 2023) that recursive improvement loops help only above a certain base-model capability — the same loop that improved results with a strong model actively degraded them with weaker ones. Recursive structure is not a substitute for intelligence.
Capacity pricing: Pricing agent work as team capacity (pod size × seniority-blended rates × duration × escalation multipliers) rather than as a scoped project with fixed fees. Aligns incentives with utilization and quality, not output count.
Challenge card: A solution-agnostic problem statement in the form "How might we…?" used in discovery workshops to surface latent needs and divergent thinking, rather than converging on pre-selected use cases.
Context assembly: The process of gathering, filtering, ranking, and formatting retrieval results, policies, decision graphs, and examples into a prompt that fits the context window and maximizes relevance. Core function of the harness.
Context budget: The available context window (e.g., 128K tokens) allocated across system prompt, user input, retrieval results, memory, and reasoning trace. Effective harnesses make trade-offs to prioritize high-signal information.
Context engineering: The discipline of optimizing what information, structure, and framing are presented to a model at invocation time to maximize correctness, safety, and efficiency. Encompasses retrieval, ranking, formatting, and precedent selection.
Context playbook: A structured, itemized alternative to the ever-lengthening prompt: context maintained as identified bullets that are incrementally added, deduplicated, and merged by deterministic logic rather than rewritten wholesale (ACE; Zhang et al. 2025). Designed to prevent context collapse and brevity bias during iterative updates.
Context graph: A persistent graph encoding an organization's decisions, policies, precedents, and actors, with edges capturing relationships like justified-by, exception-to, superseded-by, and depends-on. Stores organizational judgment, not isolated facts.
Context rot: Degradation of context quality over time as policies evolve, actors change roles, or decisions are reversed without updating the graph. A harness must include maintenance protocols to detect and repair rotten context.
Context window: The maximum number of tokens (input plus output) that a model can process in a single invocation. Frontier models now support 100K+ tokens; harness design must optimize within this hard limit.
Cross-model review: Using one model to evaluate or critique the output of another, either to catch errors the original model's priors would miss or to reduce hallucination in high-stakes tasks. Often cheaper and safer than human review for the first pass.

D

Diversity collapse: A failure mode of evolutionary and RL improvement loops in which the population degenerates into variants of one known high-reward solution. Countered with explicit mechanisms — novelty-based rejection sampling, exploration bonuses — and especially dangerous in open-ended work where the best path initially scores worse under the current evaluator.
Double Diamond: A design framework (discover, define, develop, deliver) commonly used in discovery and requirements workshops. In harness engineering, the first diamond identifies latent user needs; the second develops and hardens the agent spec.

E

Eval: A test that measures whether an agent's behavior meets a specification: correctness, safety, latency, cost, or compliance criteria. Distinguished from observability (what happened) and monitoring (does it still work?).
Eval harness: The infrastructure that runs evals: test case generation, prompt templates, model execution, scoring logic, and threshold gates. Often reused across multiple agents and models.
Eval overfitting: Optimization of agent behavior to pass a narrow eval set, often at the cost of real-world performance on out-of-distribution inputs. Prevented by diverse golden sets, regression suites, and staged readiness gates.
Evolutionary harness search: Optimizing harness or workflow designs by maintaining a population of candidates, using an LLM as the mutation operator, and selecting on eval fitness (AlphaEvolve, AFlow, Darwin Gödel Machine). Effective when candidate fitness is fast, objective, and cheap to evaluate; struggles where evaluation is slow, ambiguous, or heuristic.

F

Filesystem-as-memory: A long-horizon design pattern in which durable state — experiment logs, code diffs, error traces, past rollout trajectories — lives in files rather than the context window. Because reading, writing, and editing files is a foundational LLM skill, file-based memory automatically benefits as core model capability improves.
Fine-tuning: Retraining or adapter-tuning a model on domain-specific examples to improve task performance. In harness engineering, fine-tuning is often deferred in favor of context engineering, since strong context often outperforms weak fine-tuning at lower cost.
Forward-deployed engineer (FDE): A technical leader embedded in a business unit or customer organization to translate requirements into agent specs, shepherd rollout, and capture operational feedback for harness improvement.
Frontier model: A state-of-the-art large language model released by a leading lab, representing the current performance frontier. Harness engineering assumes deployment of frontier models, not commodity older models.

G

Golden set: A curated collection of representative test cases with ground-truth labels, used to measure agent performance on high-stakes tasks and detect regressions. Kept separate from development and early-stage eval data to prevent overfitting.
Guardrail: A constraint, filter, or veto applied to an agent's behavior before or after tool execution to enforce safety, compliance, or business rules. Examples: output content filters, rate limits, escalation triggers, and action-space boundaries.

H

Hardened spec: A requirements specification for an agent refined through discovery, prototyping, and user feedback, explicitly capturing success metrics, failure modes, escalation paths, and autonomy boundaries. Ready for harness build-out.
Harness: Everything wrapped around a model to make it dependable and measurable in production: orchestration, context assembly, memory systems, tool integration, guardrails, evaluation, observability, and unit economics. The model is the commodity; the harness is the product.
Harness engineering: The discipline of designing, building, and operating the infrastructure that makes AI agents reliable, safe, and economical. Treats the model as a component, not the solution, and focuses on context, memory, tools, guardrails, evals, and business metrics.
Human-in-the-loop (HITL): An agent design pattern in which the system flags uncertain or high-risk decisions for human review or approval before execution. Often combined with escalation thresholds and progressive autonomy.

I

Idempotency: A property of an operation: applying it multiple times produces the same result as applying it once. Idempotent agent actions simplify recovery, retry logic, and observability in distributed systems.

K

Kill switch: An operator override that immediately disables an agent's action surface, halts autonomous loops, or routes all decisions to human review. A critical safety component for production harnesses.

L

Least-capability principle: The practice of granting an agent only the minimum permissions, tool access, and context required to accomplish its task. Reduces blast radius and makes guardrails simpler and more effective.
LLM-as-judge: Using a frontier model to score the quality, safety, or correctness of another model's output in an eval. Cost-effective for subjective dimensions but can inherit the original model's biases.

M

Meta-agent: An agent whose task is to propose, modify, or create other agents, workflows, or harnesses rather than to solve end-user tasks directly (ADAS, Meta-Harness, Hyperagents). The top rung of the optimization ladder: optimizer code improving harness code.
MCP (Model Context Protocol): An open standard for connecting models to tools, data sources, and external systems. Standardizes how an agent integrates resources into its context, improving composability and reducing harness fragmentation.
Model tier: A classification of models by capability, cost, and latency (e.g., frontier, advanced, standard). A harness may route different tasks or autonomy levels to different tiers based on risk and budget.
Multi-agent system: A collection of specialized agents that collaborate or coordinate to solve problems larger than any single agent can address. Requires orchestration, message passing, and consensus protocols.

O

Observability: The ability to inspect and understand what an agent did: its inputs, reasoning steps, tool calls, context used, and outputs. Distinct from evaluation, which judges whether those actions were correct.
Orchestration: The logic that coordinates an agent's execution: sequencing tasks, routing decisions, branching based on outcomes, managing state, and handling errors. Often encoded in a state machine or workflow engine.
Orchestrator-workers pattern: A multi-agent design in which a central orchestrator decomposes tasks and coordinates specialized worker agents. Reduces hallucination and improves tractability by dividing labor.
Organizational memory: Persistent knowledge shared across an organization or harness ecosystem: policies, precedents, decision graphs, and institutional context. Distinguished from agent-specific memory and session state.

P

Per-task cost: The end-to-end cost of completing a unit of work, as opposed to per-token price. A poor correlation between the two is common: a model cheaper per token can cost more per task by working longer and reading more. Run-cost decisions require task-level measurement on your own workload, not pricing-page arithmetic.
Pilot-to-production gap: The difference between a controlled pilot environment and real-world production: volume, diversity, adversarial inputs, and edge cases often invalidate assumptions made in small-scale testing. Bridged by staged rollout and progressive autonomy.
Pod: A cross-functional team responsible for an agent or agent system in production, including engineering, product, compliance, and operations. Often treated as a capacity unit for planning and billing.
Progressive autonomy: A deployment strategy in which an agent begins with high human oversight and gradually gains autonomy as it demonstrates reliability, measured by eval scores and observability data. Reduces risk and builds stakeholder confidence.
Project pricing: Pricing agent work as a fixed-fee or scoped project with a defined endpoint. Contrasts with capacity pricing; often misaligns incentives for long-lived or evolving systems.
Prompting inversion: The frontier-era reversal in which stronger models perform better with fewer constraints, fewer examples, and more context—invalidating older rulebook-style prompting practices. Requires harness redesign to exploit.

R

RAG (retrieval-augmented generation): A technique in which an agent retrieves relevant external documents or data before generating its response, reducing hallucination and grounding answers in sources. A core component of context assembly.
Recursive self-improvement (RSI): A feedback loop in which an AI uses its current capability to improve the machinery that produces its capability (Good 1965; Yudkowsky 2008). In practice the near-term path runs through improving training pipelines and deployment harnesses rather than a model directly rewriting its own weights.
Readiness gate: An objective threshold (eval scores, regression results, escalation rates, user feedback) that must be met before an agent gains deployment, wider autonomy, or handling higher-stakes decisions. Prevents premature rollout.
Regression suite: A set of tests designed to detect performance degradation after model updates, prompt changes, or harness modifications. Distinct from golden-set evals; often includes edge cases and prior bugs.
Retrieval precision: The fraction of retrieved documents that are relevant to the agent's task, expressed as a percentage or ranked score. High precision is critical for context assembly; low precision wastes context budget and introduces noise.
Reward hacking: An optimization loop exploiting its signal rather than achieving the intended objective: overfitting unit tests, gaming a judge model, exploiting benchmark artifacts. The reason evaluators and permission systems must sit outside any loop that evolves a harness.
Review architecture: The design pattern for how decisions are checked in an agent system: who reviews (human, cross-model, self), what they review (outputs, reasoning, actions), and at what latency cost. Critical for safety-critical systems.
Run-cost: The variable cost of executing an agent invocation: model inference tokens, tool calls, retrieval queries, and orchestration overhead. Distinct from fixed harness costs and amortized development.

S

Self-improving harness: A harness whose own code is proposed, evaluated, and merged by an automated loop — typically via weakness mining, bounded edit proposal, and no-regression validation on held-in and held-out splits (Self-Harness, Darwin Gödel Machine) — under evaluation and permission control that sit outside the loop.
Self-review trap: The failure mode in which a model or person checking its own work reproduces the same priors and blind spots that produced the error; independent review (human, cross-model, or diverse) is required to catch what self-review cannot.
Session memory: Transient state retained within a single user interaction or agent invocation, typically stored in a context window or short-term buffer. Distinct from agent memory (persistent across sessions) and organizational memory (system-wide).
Spec-driven development: A development methodology in which a hardened spec is written first, evals are derived from it, and agent implementation is guided by eval results. Reduces rework and misalignment.
Subagent: A child agent spawned by a parent to pursue a hypothesis, run an experiment, or handle an isolated subtask in parallel without polluting the main context. The parent acts as a small process manager — launch, inspect logs, cancel, merge — and results should persist as files and status records, not transient chat context.
System prompt: A fixed text prepended to all model invocations that defines the agent's role, constraints, output format, and reasoning approach. Acts as the primary control surface for shaping agent behavior without fine-tuning.

T

Telemetry: Structured data emitted during agent execution: latency, token usage, tool calls, escalations, and error counts. Aggregated telemetry enables monitoring, alerting, and capacity planning.
Token economics: The study of input/output token efficiency, context reuse, and batch processing to minimize model inference costs. A frontier-model harness must optimize token spend to stay economical at scale.
Tool: An action or function an agent can invoke: API calls, database queries, file operations, or integration endpoints. Tools expand the agent's action surface but also introduce latency, cost, and error modes.
Trace: A complete record of a single agent invocation: inputs, model outputs, tool calls, results, guardrail checks, and final action. Traces enable debugging, auditing, and observability analysis.

U

Unit economics: The per-transaction or per-session cost and revenue model for an agent, typically expressed as (model + tool + orchestration cost) vs. (value captured or labor replaced). Critical for assessing viability and ROI.

W

Working state: Intermediate computation, partial results, or decision context retained during a multi-step agent task. Must be tracked in observability and often stored separately from session or organizational memory.