Pilot to Production: The Forward-Deployed Craft
Ask almost any enterprise how many AI pilots they've run in the last two years and the number is large — a dozen, two dozen, sometimes more, spread across claims, field ops, sales, marketing, HR. Ask how many of those are running in production, serving real transactions, with an owner and a budget line, and the number collapses. Surveys of enterprise AI adoption have reported that roughly a third of organizations have any agent running in production at all, despite the overwhelming majority having piloted something. That gap — pilots everywhere, production nowhere — is the defining fact of enterprise AI in 2026, and it is not a gap in model capability. It is a gap in engineering discipline, and this lesson is about the specific craft that closes it.
Autopsy of a dead pilot
Pilots don't usually die of a dramatic failure. They die quietly, of neglect, roughly six to nine months after a promising demo, when the person who championed them moves to a different priority and nobody notices the thing was never actually finished. Look closely at the corpses and you find the same four missing organs, in some combination, almost every time.
No evals. Nobody ever defined, in a measurable way, what "good" looked like for this specific workflow. The pilot was judged by whether the demo impressed a steering committee, not by performance against a labeled set of real cases. When someone eventually asks "is this ready for real customers," there is no answer, because there was never a question anyone could answer with a number.
No owner. The pilot belonged to an innovation team, a digital transformation office, or an enthusiastic product manager's twenty-percent time. Production belongs to whoever's on call when it breaks at 2 a.m., whoever owns the budget line when finance asks what it costs to run, and whoever's name is on the escalation path when a customer complains. Nobody in the innovation team signed up to be that person, and nobody in the operating business volunteered, because the operating business didn't build it and doesn't trust it yet.
No economics. Nobody priced the run-cost per transaction at real volume, and nobody priced the value of getting the transaction right versus a human doing it. The pilot ran on a demo budget with a handful of test cases. Nobody modeled what it costs when a frontier model call happens on every one of 40,000 monthly claims, or what a 2% error rate at that volume actually costs the business in rework and complaints. The business case, when someone finally asks for one, doesn't exist.
Demo scope, not production scope. The demo ran on ten curated examples, a synthetic dataset, or last quarter's already-resolved cases — not the org's live, messy, real data with its missing fields, contradictory records, and edge cases. It never touched the identity and access management system, never went through security review, never got anywhere near procurement, because a demo doesn't need to. Production does, and the distance between "works in the demo" and "cleared security review, integrated with the real systems, approved by procurement, running on real data" turns out to be most of the actual engineering work — work the pilot never did because nothing forced it to.
Pilots die from missing evals, missing owners, and missing economics — not missing intelligence. An organization can have access to the most capable model on the market and still fail to reach production, because the barrier was never the model's reasoning ability. It was the absence of a way to measure success, a person accountable for the outcome, and a number that says the thing is worth running at scale.
The forward-deployed craft
The role that closes this gap has gone by different names — forward-deployed engineer, solutions architect, embedded technical lead — but the job is consistent regardless of title: someone who sits inside the customer's actual environment, with actual access to actual data and actual stakeholders, and does the unglamorous work of turning a demonstrated capability into a system the business will actually run and depend on. It is not a research role and not a sales role. It is closer to a general contractor who shows up after the architect's drawings are done and has to reconcile them with the actual plumbing behind the actual walls.
Here is what a representative Tuesday looks like for someone doing this work well, for a mid-sized insurer trying to move a claims-triage agent from pilot to production.
Morning: reads the client's data reality before believing any architecture diagram. The proposed architecture assumes claims documents arrive as structured JSON. The forward-deployed engineer pulls fifty real claims from the production system and finds that a third arrive as scanned PDFs with inconsistent formatting, another chunk come through an adjuster's free-text notes field, and a handful reference a legacy policy system that isn't in the data dictionary anyone handed over. The clean architecture diagram from the kickoff deck describes a system that doesn't match the data the business actually has. This gets found by looking, not by asking — the people who could have told you in a meeting often don't know either, because nobody has looked at fifty real cases end to end in years.
Mid-morning: hardens specs against how the org actually makes decisions. The written policy says claims under $5,000 with clean documentation auto-approve. Watching three adjusters actually work for an hour reveals that experienced adjusters also cross-check a specific fraud-pattern flag that isn't written down anywhere, because it was passed down verbally from a supervisor two years ago and never made it into the policy manual. If the agent's spec is built from the written policy alone, it will miss a behavior every experienced human in the workflow considers table stakes. The forward-deployed engineer's job is to surface this gap before it becomes a production incident, not after.
Midday: builds the eval set with the workflow's owners, not for them. Rather than assembling a labeled eval set alone and presenting it as done, the engineer sits with two senior adjusters and works through eighty real historical claims together, capturing not just the right answer but the reasoning an experienced adjuster used to get there — including the fraud-pattern flag from the morning. This eval set becomes the thing production readiness is measured against later, and because the adjusters built it with the engineer, they trust it and will defend its verdicts to their own management, instead of dismissing it as something IT cooked up in isolation.
Afternoon: navigates procurement and security review as design constraints, not obstacles. Security wants to know where claims data physically resides and whether it leaves the org's network boundary during model inference. Procurement wants to know the total cost structure, not just the per-call API price. Rather than treating these as bureaucratic friction to route around, the engineer treats the answers as inputs to the actual system design — data residency requirements shape which model deployment option is viable; procurement's cost ceiling shapes whether a frontier model runs on every claim or only on the ones a cheaper triage step flags as ambiguous. Design decisions made in ignorance of these constraints get rebuilt later, at higher cost, once compliance finds out.
Late afternoon: translates model behavior for executives at the altitude they need. The VP of claims doesn't want to hear about retrieval-augmented generation or context windows. She wants to know: what happens when the agent is wrong, how often is it wrong, who catches it, and what does it cost if it's wrong on a claim that matters. The engineer translates the eval results, the human-in-the-loop design, and the cost model into that language — not dumbing it down, but converting engineering detail into the decision-relevant facts an executive actually needs to sign off responsibly.
None of this Tuesday involved writing a novel algorithm or fine-tuning a model. It involved reading real data, watching real people work, building trust with the people whose workflow is being changed, and translating between engineering reality and business and compliance constraints. This is most of what closes the pilot-to-production gap, and it is systematically undervalued by organizations that think the hard part is the model.
What the role is not
It helps to define this role by contrast with three adjacent roles it gets confused with, because the confusion causes organizations to staff the gap with the wrong skill set.
It is not a sales engineer. A sales engineer's job is to make the demo work well enough to close a deal, on a timeline measured in weeks, optimized for impressiveness under controlled conditions. The forward-deployed engineer's job starts after the deal closes and doesn't end until the system is running unattended in production, optimized for correctness under the org's actual messy conditions. Confusing the two means treating a demo as a finish line instead of a starting point.
It is not a consultant. A consultant, in the pejorative sense the term has earned in this context, produces a recommendation, a slide deck, and a departure — leaving the organization to implement the finding, often after the political capital for change has cooled. The forward-deployed engineer ships. The output is a running system with an eval suite, a cost model, and an owner, not a document describing what a running system would ideally look like.
It is not a product engineer. A product engineer builds a general-purpose capability meant to work across many customers with configuration, not customization — the discipline is abstraction and reusability. The forward-deployed engineer lives inside one customer's specific environment, with that customer's specific data quirks, that customer's specific compliance regime, and no illusion that the solution generalizes cleanly to the next customer without real rework. Both roles matter and a mature organization needs both, but putting a product engineer's instincts (build the general case, avoid one-off logic) onto forward-deployed work produces elegant systems that don't fit the one real environment they need to run in.
Organizations sometimes try to solve the pilot-to-production gap by hiring more of the wrong role — another sales engineer to run more polished demos, another strategy consultant to produce another readiness assessment. Both add motion without closing the gap, because neither role is accountable for a running system with a named owner, a priced run-cost, and an eval-verified success criterion. If your pilot-to-production initiative's headcount plan doesn't include someone who will still be answerable for the system's behavior a year from now, you have not staffed the gap — you have staffed another pilot.
Discovery done right: challenge statements before use cases
The forward-deployed craft starts before any architecture gets drawn, in discovery — and this is where most enterprise AI initiatives make their first and most consequential mistake. The common pattern: someone arrives with a pre-converged use-case list — "GenAI does X, GenAI does Y" — skipping the actual problem-definition work and jumping straight to a solution shape. This produces automation theater: technically functioning systems that automate a step nobody confirmed was the right step to automate, aimed at a version of the problem nobody actually validated with the people living it.
Discovery done right produces solution-agnostic challenge statements instead — "How might we…" framings that name a problem without presupposing an answer. "How might we reduce the volume of after-the-fact purchase orders that never should have been discretionary in the first place?" is a challenge statement. "Build a GenAI agent that auto-generates purchase order justifications" is a pre-converged use case that may or may not address the actual problem, and frequently doesn't, because it was written by someone excited about a capability rather than someone who diagnosed a workflow failure.
The practical version of this: challenge statements get written on cards, generated from structured interviews and shadowing sessions with the people who actually do the work, then clustered on a wall by theme. This is Double Diamond thinking applied to enterprise AI discovery — a divergent phase that deliberately generates more problem framings than you need, followed by a convergent phase that narrows to the two or three challenge statements with the highest combination of business impact and organizational readiness to change. Only after convergence does a solution shape — agent, workflow redesign, simple automation, or sometimes no technology at all — get proposed, and it gets proposed against a validated problem instead of a hunch.
Discovery should produce solution-agnostic challenge statements, not pre-converged use-case lists. A use-case list skips the step where you find out whether you're automating the right thing; a challenge statement forces that step to happen before any architecture gets drawn. The cost of skipping it is a technically competent system built for the wrong problem — which looks, for months, exactly like a good pilot.
The deployment checklist
By the time a pilot is a candidate for production, the forward-deployed engineer should be able to check every row of a short, unglamorous list — and unable to, honestly, is the signal that it isn't ready, regardless of how good the demo looked.
| Gate | What "done" looks like |
|---|---|
| Eval gate | A labeled eval set built with the workflow's actual owners, a defined passing threshold, and a current score above it |
| Owner named | A specific person or team in the operating business, not the innovation function, accountable for the system's behavior post-launch |
| Run-cost priced | Per-transaction cost modeled at real production volume, compared against the value of a correct outcome and the cost of an error |
| Rollback plan | A tested, documented way to disable the agent and fall back to the prior process without a multi-day scramble |
| HITL placement | Human-in-the-loop checkpoints placed at the specific failure modes the eval revealed, not everywhere or nowhere by default |
| Audit trail | Every decision the agent makes is logged with enough context to reconstruct why, for a regulator, a customer complaint, or an internal review |
| Support model | A defined on-call and escalation path for when the system misbehaves in production, staffed by people who understand the harness, not just the model |
Take a pilot in your organization that's been "almost ready to go live" for more than one quarter. Run it against the seven-row table above with the people closest to it, honestly, gate by gate. Most stalled pilots fail on three or four rows at once, usually eval gate, owner, and economics together — which is consistent with the autopsy pattern in this lesson. Write down which rows fail and who would need to do what to close each one. That list is the actual remaining project plan; the pilot's original plan, whatever it said, was apparently not it.
The license-versus-deployment line
An agent platform purchase without engineering is a license, not a deployment. Buying access to a capable model or a polished agent-building platform gets an organization a tool, in the same sense that buying a CRM license gets an organization a tool — the value only materializes when someone configures it against real workflows, real data, and real accountability structures, and that configuration work is exactly the harness engineering and forward-deployed craft this lesson and this course have described. Vendors selling agent platforms are, correctly, selling a component. Organizations that treat the purchase order as the finish line are buying a very expensive piece of shelfware with an API.
The organizational pattern that actually works, across the insurers, utilities, and marketing organizations that have made this transition successfully, is consistent: a platform team that owns the shared harness infrastructure — the evaluation framework, the tool contracts, the logging and telemetry standards, the review architecture from lesson 13 — paired with engineers embedded directly in the operating business units, close enough to the claims adjusters and dispatch coordinators to do the Tuesday-in-the-life work this lesson described. Neither half works alone. A platform team with no embedded presence in the business builds elegant infrastructure nobody's workflow actually needs. Embedded engineers with no shared platform reinvent evaluation and guardrail infrastructure badly, one business unit at a time. Together, they are the organizational structure that turns a license into a deployment.
Key takeaways
- Enterprise AI's defining gap is pilots-to-production, not capability: most organizations have piloted an agent, reported figures suggest only around a third have any agent running in production.
- Dead pilots share four missing organs: no evals (no measurable definition of good), no owner (belongs to innovation, not the operating business), no economics (run-cost and value never priced), and demo scope instead of production scope (never touched real data, security, or procurement).
- The forward-deployed craft is reading real data before trusting architecture diagrams, hardening specs against how the org actually decides, building evals with workflow owners, treating procurement and security as design constraints, and translating model behavior for executives.
- This role is distinct from a sales engineer (doesn't stop at the demo), a consultant (ships instead of recommending and leaving), and a product engineer (lives inside one customer's specific mess rather than building the general case).
- Discovery should produce solution-agnostic "How might we" challenge statements clustered through divergence before convergence — not a pre-converged use-case list that skips problem definition and produces automation theater.
- A seven-gate deployment checklist — eval gate, owner, run-cost, rollback, HITL placement, audit trail, support model — is a practical test for whether a pilot is actually ready, regardless of demo quality.
- An agent platform purchase without engineering is a license, not a deployment; the organizational pattern that works pairs a platform team owning shared harness infrastructure with engineers embedded in the business.
Check your understanding
1. According to the lesson's autopsy, what most commonly kills an enterprise AI pilot before it reaches production?
2. How does the lesson distinguish the forward-deployed engineer from a sales engineer?
3. What is the key difference between a solution-agnostic challenge statement and a pre-converged use-case list in discovery?
4. Per the lesson's closing argument, what organizational structure actually closes the pilot-to-production gap at scale?