Context Engineering: What the Model Sees

Reading time ~12 min · Quiz at the end

A phrase has been going around industry conferences and vendor blog posts for the last couple of years: "context engineering replaced prompt engineering." It's a catchy line, and it's pointing at something real, but it's not quite accurate, and the inaccuracy matters if you're the one building the system. Context engineering didn't replace prompt engineering the way a new tool replaces an old one. It's a different, larger layer of the stack that happened to become visible once teams got past writing clever instructions and started noticing that the instructions weren't the problem — everything else in the model's input window was.

Context engineering is the discipline of designing what the model sees on every single inference call. Not what you'd like it to see, not what's theoretically available in your data warehouse — what actually lands in the token sequence the model reads before it generates a response. Prompt engineering, by contrast, is really about how you phrase the instructions within that window. You can have beautifully phrased instructions sitting inside a context window stuffed with three redundant copies of the customer's account history, two stale product descriptions, and a wall of boilerplate legal text — and the phrasing won't save you, because the model's attention is a finite resource being spent on the wrong things.

Context budgets are real budgets

Every model has a maximum context window, and modern frontier models offer windows large enough that teams stop thinking about the limit at all — which is exactly the mistake. A context window is not free just because it's large. Every token you put in front of the model does two things: it costs money and latency (you pay per token, and processing time scales with input length), and it competes for the model's attention against every other token in the window. A 200,000-token context window that could technically fit your entire customer history is not an invitation to put your entire customer history in it. Treat context as a budget with a return-on-investment question attached to every line item: does this specific piece of information increase the odds of a correct output for this specific call, enough to justify its cost in tokens and in attention dilution? A claims adjuster's entire five-year interaction history is mostly irrelevant to whether this specific claim should fast-track. The three prior claims that were denied for a similar reason are highly relevant. Budget-conscious context assembly finds the second set and leaves the first set in the database, not the prompt.

The cost consequences of this are not marginal, and they are decided by the harness, not the model. Databricks (2026) ran the same model at the same thinking effort through different agent harnesses and found the per-task cost differed by more than 2x at unchanged output quality. The driver was context fed per turn: the cheaper harness sent roughly 3x less context on each turn, kept a tighter working set, and finished the task in fewer turns. Same model, same quality, twice the bill — and the only variable was how disciplined the harness was about what it put in front of the model each turn.

The lesson to draw is that context discipline is not merely a quality lever, it is the dominant cost lever, and at enterprise scale the two are the same conversation. "What the model sees" is not decided by the model; it is decided by the harness, on every single turn, and every undisciplined turn compounds — more tokens now, more turns to finish, more cost across the whole trajectory. A team that treats the context window as free because it is large is not just risking accuracy, it is quietly paying twice for the same result.

Retrieval precision versus recall — the trade-off you can't avoid

Any retrieval-augmented step — pulling documents, past cases, policy text, or prior tickets into context — sits on a spectrum between precision (only relevant results, but you might miss something) and recall (catch everything possibly relevant, but you'll pull in noise). Enterprise teams building their first retrieval pipeline almost always default toward recall: "let's cast a wide net so we don't miss anything important." This feels safe and is usually wrong, because the cost of the noise is not neutral — it's context rot, diluted attention, and in the worst case, the model citing the wrong one of five similar-looking retrieved documents as authoritative. The better default, especially for anything with legal, financial, or safety consequences, is to bias toward precision and treat recall gaps as a retrieval-quality problem to fix at the source — better metadata, better chunking, better query formulation — rather than a problem to paper over by pulling in more results "just in case." A utility field-operations agent answering "what's the isolation procedure for this transformer model" needs the one correct procedure document, not the top eight semantically similar documents where three are for a different transformer generation entirely.

This four-tier model is useful because each tier has a different engineering discipline attached to it. System doctrine should be version-controlled and reviewed like code, because it's shared across every single call your system makes — a bad change here has blast radius across the entire agent. Task context should be templated per workflow type, not hand-written per call. Retrieved knowledge is where your retrieval pipeline's precision-recall tuning lives, and it's the tier most exposed to the anti-patterns below. Working state needs active pruning logic — deciding what from the conversation so far is still relevant and what can be summarized or dropped — because it's the tier most likely to grow unbounded if nobody owns it.

Four anti-patterns worth naming

Dump-everything RAG. The retrieval step returns the top twenty semantically similar chunks "to be safe," most of them marginally relevant, and all of them consuming budget and attention. The fix is tightening the retrieval query and the similarity threshold, and accepting that occasionally missing a marginal result is a better trade than routinely drowning the model in noise. Stale summaries. A long-running case gets periodically summarized to save tokens, but the summarization step itself uses a weak prompt or an outdated summary-of-a-summary chain, and critical details quietly drop out three summarization cycles in. A customer's stated allergy, a previously agreed exception to standard policy, or a flagged fraud concern can vanish this way, silently, with no error message — the summary just doesn't mention it anymore. Unbounded conversation history. The simplest and most common anti-pattern: every turn of a long-running interaction gets appended to context with no pruning strategy at all, until either the context window fills up or the sheer bulk of earlier, now-irrelevant turns degrades the model's attention on the current ask. Duplicated instructions. System doctrine, task context, and a retrieved document all separately state a version of the same policy, sometimes with subtle wording differences introduced when someone updated one copy and not the others. The model now has to adjudicate between three slightly different statements of "the same" rule, and which one it follows becomes effectively arbitrary.

Anti-pattern check for a context assembly pipeline:

  [ ] Does retrieval return a bounded, precision-tuned set,
      or "everything above some low similarity threshold"?
  [ ] Is there an explicit summarization step with its own
      quality check, or does context just get truncated?
  [ ] Is there a pruning policy for working state, or does
      the conversation array grow every turn forever?
  [ ] Does policy text exist in exactly one place, referenced
      everywhere else, or is it copy-pasted into multiple tiers?

The rule: every token earns its place

If there is one operating principle to take from this lesson, it's this: every token in context is a claim on the model's attention, and that claim has to be justified. This reframes context assembly from an additive exercise ("what else should we include to be thorough") to a subtractive one ("what can we remove without losing anything the model needs to get this right"). Teams that internalize the subtractive framing consistently end up with smaller, faster, cheaper, and more accurate context pipelines than teams that default to inclusion. This is counterintuitive to engineers trained on the assumption that more information is strictly better — with a probabilistic reasoner reading a finite window, it usually isn't.

You cannot do this without evals

Here is the uncomfortable dependency this entire lesson has been building toward: none of the trade-offs above — precision versus recall, how aggressively to prune working state, whether a given retrieved document is helping or hurting — can be tuned by intuition alone. You need a way to measure, on a representative set of real cases, whether a change to context assembly made outputs better or worse. Without that measurement, every context engineering decision is a guess dressed up as an opinion, and two experienced engineers will confidently disagree about whether adding or removing a given piece of context helped. This is why lesson 11 (observability versus evals) and lesson 12 (building eval harnesses) exist, and why they matter as much as this lesson does. Context engineering supplies the levers; evals supply the feedback that tells you which way to pull them. A team with excellent context-engineering instincts and no eval suite will still ship regressions, because instinct doesn't scale past the cases the engineer happens to remember checking by hand.

Context as an evolving playbook

The subtractive discipline above treats a context payload as something you assemble carefully for each call. But the best systems are starting to make a further move: rather than hand-tuning context assembly forever, they let the context itself accumulate and refine over time as a structured artifact — closer to a maintained playbook than a prompt you keep rewriting. The instinct most teams have when a context strategy underperforms is to lengthen the prompt: add another instruction, another example, another caveat. That is how you get the bloated, self-contradicting windows this lesson has warned against. Agentic Context Engineering (ACE; Zhang et al., 2025) proposes the opposite discipline — maintain a single context playbook of itemized bullets, each with a stable identifier and a description, and evolve it through incremental edits rather than wholesale rewrites.

ACE splits the work of maintaining that playbook across three roles. A Generator runs tasks and produces trajectories that reference the playbook's bullets by identifier, so you can see which guidance actually gets used. A Reflector reads those trajectories — successful and failed alike, because a failure that traces to a missing or wrong bullet is as informative as a success — and distills what it learns into candidate insights. A Curator then emits incremental itemized entries that deterministic merge logic folds into the playbook, never rewriting the whole blob, and entries are periodically deduplicated and refined. The point of the structure is not novelty for its own sake; it is to prevent two specific failure modes of naive iterative rewriting — "context collapse," where repeated full rewrites erode accumulated detail into vague summary, and "brevity bias," where each rewrite trims toward shorter text and quietly drops the specifics that made the context useful.

One level up sits Meta Context Engineering (MCE; Ye et al., 2026), which separates the mechanism of context management from the content being managed. It runs a bi-level optimization: an outer loop evolves "skills" — the context-management strategies themselves, instantiated as files in a directory, a static skill.md plus dynamic operators for searching, selecting, filtering, and formatting context — against validation performance, while an inner loop optimizes the actual task context under the current skill against training performance. The outer loop learns how to manage context; the inner loop applies that learned strategy to the case at hand. It is the same freeze-and-isolate discipline the improvement loop (lesson 14) will formalize, turned recursively on context assembly itself.

Where does this end? Weng (2026) makes a forward-looking argument worth sitting with: humans maintain memory and context over a lifetime without an external engineer curating it, so context engineering will and should migrate over time from the software layer toward core model intelligence — the model itself getting better at deciding what to remember, retrieve, and attend to. But she is careful about what that does and does not eliminate. The need to specify goals, constraints, context, and evaluation criteria does not disappear when the plumbing improves; it just moves. The analogy is prompt engineering itself: the manual bag of tricks — "let's think step by step," rigid output templates, elaborate role-play framing — faded as instruction-tuning got better, yet the underlying act of specifying what you want never went away. Expect the same arc here. The mechanics of context assembly will increasingly be handled for you; the responsibility for saying what the system is for will not.

What comes after per-call context

Everything in this lesson concerns what a single inference call sees. But organizations accumulate judgment that outlives any single call or even any single case — an underwriter's accumulated sense of which claim patterns correlate with fraud, a field-operations lead's knowledge of which crew combinations work well together, a sales team's evolving understanding of which discount justifications actually hold up. That judgment is not per-call context; it's a persistent structure that per-call context should be able to draw from. Lesson 5 covers how to represent that as a context graph — the layer above the one this lesson describes, and the one that starts to look less like prompt assembly and more like organizational memory made queryable.