Context Engineering: What the Model Sees

Reading time ~15 min · Last reviewed Sep 2026 · Quiz at the end

A phrase has been going around industry conferences and vendor blog posts for the last couple of years: "context engineering replaced prompt engineering." It's a catchy line, and it's pointing at something real, but it's not quite accurate, and the inaccuracy matters if you're the one building the system. Context engineering didn't replace prompt engineering the way a new tool replaces an old one. It's a different, larger layer of the stack that happened to become visible once teams got past writing clever instructions and started noticing that the instructions weren't the problem — everything else in the model's input window was.

Context engineering is the discipline of designing what the model sees on every single inference call. Not what you'd like it to see, not what's theoretically available in your data warehouse — what actually lands in the token sequence the model reads before it generates a response. Prompt engineering, by contrast, is really about how you phrase the instructions within that window. You can have beautifully phrased instructions sitting inside a context window stuffed with three redundant copies of the customer's account history, two stale product descriptions, and a wall of boilerplate legal text — and the phrasing won't save you, because the model's attention is a finite resource being spent on the wrong things.

Context budgets are real budgets

Every model has a maximum context window, and modern frontier models offer windows large enough that teams stop thinking about the limit at all — which is exactly the mistake. A context window is not free just because it's large. Every token you put in front of the model does two things: it costs money and latency (you pay per token, and processing time scales with input length), and it competes for the model's attention against every other token in the window. A 200,000-token context window that could technically fit your entire customer history is not an invitation to put your entire customer history in it. Treat context as a budget with a return-on-investment question attached to every line item: does this specific piece of information increase the odds of a correct output for this specific call, enough to justify its cost in tokens and in attention dilution? A claims adjuster's entire five-year interaction history is mostly irrelevant to whether this specific claim should fast-track. The three prior claims that were denied for a similar reason are highly relevant. Budget-conscious context assembly finds the second set and leaves the first set in the database, not the prompt.

The cost consequences of this are not marginal, and they are decided by the harness, not the model. Databricks (2026) ran the same model at the same thinking effort through different agent harnesses and found the per-task cost differed by more than 2x at unchanged output quality. The driver was context fed per turn: the cheaper harness sent roughly 3x less context on each turn, kept a tighter working set, and finished the task in fewer turns. Same model, same quality, twice the bill — and the only variable was how disciplined the harness was about what it put in front of the model each turn.

The lesson to draw is that context discipline is not merely a quality lever, it is the dominant cost lever, and at enterprise scale the two are the same conversation. "What the model sees" is not decided by the model; it is decided by the harness, on every single turn, and every undisciplined turn compounds — more tokens now, more turns to finish, more cost across the whole trajectory. A team that treats the context window as free because it is large is not just risking accuracy, it is quietly paying twice for the same result.

Retrieval precision versus recall — the trade-off you can't avoid

Any retrieval-augmented step — pulling documents, past cases, policy text, or prior tickets into context — sits on a spectrum between precision (only relevant results, but you might miss something) and recall (catch everything possibly relevant, but you'll pull in noise). Enterprise teams building their first retrieval pipeline almost always default toward recall: "let's cast a wide net so we don't miss anything important." This feels safe and is usually wrong, because the cost of the noise is not neutral — it's context rot, diluted attention, and in the worst case, the model citing the wrong one of five similar-looking retrieved documents as authoritative. The better default, especially for anything with legal, financial, or safety consequences, is to bias toward precision and treat recall gaps as a retrieval-quality problem to fix at the source — better metadata, better chunking, better query formulation — rather than a problem to paper over by pulling in more results "just in case." A utility field-operations agent answering "what's the isolation procedure for this transformer model" needs the one correct procedure document, not the top eight semantically similar documents where three are for a different transformer generation entirely.

This four-tier model is useful because each tier has a different engineering discipline attached to it. System doctrine should be version-controlled and reviewed like code, because it's shared across every single call your system makes — a bad change here has blast radius across the entire agent. Task context should be templated per workflow type, not hand-written per call. Retrieved knowledge is where your retrieval pipeline's precision-recall tuning lives, and it's the tier most exposed to the anti-patterns below. Working state needs active pruning logic — deciding what from the conversation so far is still relevant and what can be summarized or dropped — because it's the tier most likely to grow unbounded if nobody owns it. How to prune it — when to clear old tool results, when to summarize, what a compaction trigger looks like, and how to keep summaries from going stale — is a lifecycle question, and lesson 6 is devoted to it. This lesson is about what belongs in the window; lesson 6 is about how the window changes over a long run.

Four anti-patterns worth naming

Dump-everything RAG. The retrieval step returns the top twenty semantically similar chunks "to be safe," most of them marginally relevant, and all of them consuming budget and attention. The fix is tightening the retrieval query and the similarity threshold, and accepting that occasionally missing a marginal result is a better trade than routinely drowning the model in noise. Stale summaries. A long-running case gets periodically summarized to save tokens, but the summarization step itself uses a weak prompt or an outdated summary-of-a-summary chain, and critical details quietly drop out three summarization cycles in. A customer's stated allergy, a previously agreed exception to standard policy, or a flagged fraud concern can vanish this way, silently, with no error message — the summary just doesn't mention it anymore. Unbounded conversation history. The simplest and most common anti-pattern: every turn of a long-running interaction gets appended to context with no pruning strategy at all, until either the context window fills up or the sheer bulk of earlier, now-irrelevant turns degrades the model's attention on the current ask. Duplicated instructions. System doctrine, task context, and a retrieved document all separately state a version of the same policy, sometimes with subtle wording differences introduced when someone updated one copy and not the others. The model now has to adjudicate between three slightly different statements of "the same" rule, and which one it follows becomes effectively arbitrary.

Stale summaries and unbounded history are named here because they are assembly failures you can spot in a single payload. The machinery that prevents them — compaction policies, pruning oversized tool results before summarizing, keeping an append-only log so a bad summary can be regenerated — is covered in lesson 6.

Anti-pattern check for a context assembly pipeline:

  [ ] Does retrieval return a bounded, precision-tuned set,
      or "everything above some low similarity threshold"?
  [ ] Is there an explicit summarization step with its own
      quality check, or does context just get truncated?
  [ ] Is there a pruning policy for working state, or does
      the conversation array grow every turn forever?
  [ ] Does policy text exist in exactly one place, referenced
      everywhere else, or is it copy-pasted into multiple tiers?

The rule: every token earns its place

If there is one operating principle to take from this lesson, it's this: every token in context is a claim on the model's attention, and that claim has to be justified. This reframes context assembly from an additive exercise ("what else should we include to be thorough") to a subtractive one ("what can we remove without losing anything the model needs to get this right"). Teams that internalize the subtractive framing consistently end up with smaller, faster, cheaper, and more accurate context pipelines than teams that default to inclusion. This is counterintuitive to engineers trained on the assumption that more information is strictly better — with a probabilistic reasoner reading a finite window, it usually isn't.

You cannot do this without evals

Here is the uncomfortable dependency this entire lesson has been building toward: none of the trade-offs above — precision versus recall, how aggressively to prune working state, whether a given retrieved document is helping or hurting — can be tuned by intuition alone. You need a way to measure, on a representative set of real cases, whether a change to context assembly made outputs better or worse. Without that measurement, every context engineering decision is a guess dressed up as an opinion, and two experienced engineers will confidently disagree about whether adding or removing a given piece of context helped. This is why lesson 17 (observability versus evals) and lesson 18 (building eval harnesses) exist, and why they matter as much as this lesson does. Context engineering supplies the levers; evals supply the feedback that tells you which way to pull them. A team with excellent context-engineering instincts and no eval suite will still ship regressions, because instinct doesn't scale past the cases the engineer happens to remember checking by hand.

Context as an evolving playbook

The subtractive discipline above treats a context payload as something you assemble carefully for each call. But the best systems are starting to make a further move: rather than hand-tuning context assembly forever, they let the context itself accumulate and refine over time as a structured artifact — closer to a maintained playbook than a prompt you keep rewriting. The instinct most teams have when a context strategy underperforms is to lengthen the prompt: add another instruction, another example, another caveat. That is how you get the bloated, self-contradicting windows this lesson has warned against. Agentic Context Engineering (ACE; Zhang et al., 2025) proposes the opposite discipline — maintain a single context playbook of itemized bullets, each with a stable identifier and a description, and evolve it through incremental edits rather than wholesale rewrites.

ACE splits the work of maintaining that playbook across three roles. A Generator runs tasks and produces trajectories that reference the playbook's bullets by identifier, so you can see which guidance actually gets used. A Reflector reads those trajectories — successful and failed alike, because a failure that traces to a missing or wrong bullet is as informative as a success — and distills what it learns into candidate insights. A Curator then emits incremental itemized entries that deterministic merge logic folds into the playbook, never rewriting the whole blob, and entries are periodically deduplicated and refined. The point of the structure is not novelty for its own sake; it is to prevent two specific failure modes of naive iterative rewriting — "context collapse," where repeated full rewrites erode accumulated detail into vague summary, and "brevity bias," where each rewrite trims toward shorter text and quietly drops the specifics that made the context useful.

One level up sits Meta Context Engineering (MCE; Ye et al., 2026), which separates the mechanism of context management from the content being managed. It runs a bi-level optimization: an outer loop evolves "skills" — the context-management strategies themselves, instantiated as files in a directory, a static skill.md plus dynamic operators for searching, selecting, filtering, and formatting context — against validation performance, while an inner loop optimizes the actual task context under the current skill against training performance. The outer loop learns how to manage context; the inner loop applies that learned strategy to the case at hand. It is the same freeze-and-isolate discipline the improvement loop (lesson 21) will formalize, turned recursively on context assembly itself.

Where does this end? Weng (2026) makes a forward-looking argument worth sitting with: humans maintain memory and context over a lifetime without an external engineer curating it, so context engineering will and should migrate over time from the software layer toward core model intelligence — the model itself getting better at deciding what to remember, retrieve, and attend to. But she is careful about what that does and does not eliminate. The need to specify goals, constraints, context, and evaluation criteria does not disappear when the plumbing improves; it just moves. The analogy is prompt engineering itself: the manual bag of tricks — "let's think step by step," rigid output templates, elaborate role-play framing — faded as instruction-tuning got better, yet the underlying act of specifying what you want never went away. Expect the same arc here. The mechanics of context assembly will increasingly be handled for you; the responsibility for saying what the system is for will not.

Just-in-time context: fetch, don't stuff

Everything so far has assumed the harness decides, before the call, what goes into the window. That is the retrieval-pipeline mental model, and for a single-shot RAG answer it is still right. Agents change the picture, because an agent can go and get context itself. Give it a way to list files, search a codebase, run a query, or open a document, and the question shifts from "what should we pre-load?" to "what is the smallest thing we must pre-load so the agent knows where to look?"

Anthropic (2025) calls this just-in-time context. Instead of pre-processing every relevant document into the prompt, the agent holds "lightweight identifiers" (file paths, stored queries, links) and loads the underlying data with tools at the moment it needs it. The same post describes the result as progressive disclosure: the agent assembles its understanding layer by layer through exploration, keeping only what it currently needs in working memory. It also names the pragmatic middle ground most coding agents use: Claude Code loads its project memory file up front for speed, then relies on primitives like glob and grep to pull in files just in time rather than indexing the repository into the prompt.

Agent Skills are the same idea packaged as a format. Per Anthropic's Skills post (2025), only each skill's name and description sit in the system prompt at startup; the full SKILL.md is read into context only when the model decides the skill applies; and any bundled reference files or scripts are opened only if the task calls for them. Three levels, each loaded on demand, so the amount of knowledge you can make available is no longer bounded by the window. Lesson 11 covers Skills as an action surface. OpenAI's Codex team arrived at the same shape for repository knowledge, keeping a short AGENTS.md as a map to deeper docs rather than a manual; lesson 9 takes that up in detail.

# Pre-stuffed: the harness guesses, every turn pays for the guess
context = system_doctrine + task + top_k(retrieve(query), k=20) + history

# Just-in-time: a small map up front, tools to fetch the rest
context = system_doctrine + task + project_map   # ~100 lines: what exists, where
tools   = [list_dir, grep, read_file(path, lines), search_tickets(query)]
# The agent reads the map, then pulls only the three files and one
# ticket this step needs. What it fetched is visible in the trace,
# so an eval can check whether it fetched the right things.

Just-in-time context is not free. Exploration costs turns and latency, and an agent with poor tools or a misleading map will wander, read the wrong file, and reason confidently from it. The trade is a precision-versus-recall decision moved from the retrieval pipeline into the agent's own behavior: you are betting that the model plus good tools will find the right three documents more reliably than your similarity threshold would. For frontier models on well-organized sources, that bet increasingly pays. For weaker models, or for sources with no navigable structure, a well-tuned pre-loaded set may still win. As with everything in this lesson, the eval suite decides.

Lessons from Manus, briefly

The most-cited practitioner write-up on agent context is Manus (Ji, 2025). Four of its lessons bear directly on this one. It treats KV-cache hit rate as the single most important production metric, citing a roughly 10x price gap between cached and uncached input tokens, which is the cost argument from the start of this lesson seen from the serving side. It uses the filesystem as context: large observations go to files, and the window keeps only the path, so anything dropped can be restored. It uses recitation, having the agent rewrite a todo file as it works so the current objective stays near the end of the window where attention is strongest. And it keeps errors in context rather than cleaning them up, because the model cannot adapt to a failure it never sees. The cache, append-only, and compaction mechanics behind these are the subject of lesson 6; the point to take here is that "what the model sees" includes its own mistakes and its own notes, deliberately.

What comes after per-call context

Everything in this lesson concerns what a single inference call sees. But organizations accumulate judgment that outlives any single call or even any single case — an underwriter's accumulated sense of which claim patterns correlate with fraud, a field-operations lead's knowledge of which crew combinations work well together, a sales team's evolving understanding of which discount justifications actually hold up. That judgment is not per-call context; it's a persistent structure that per-call context should be able to draw from. Lesson 7 covers how to represent that as a context graph — the layer above the one this lesson describes, and the one that starts to look less like prompt assembly and more like organizational memory made queryable.