Observability for Forward-Deployed Engineers Shipping AI Agents (2026)
You build the agent at the customer and then you leave. How to keep seeing what it does, prove it works, and debug it without access.
On this page
Forward-deployed engineers (FDEs) build AI systems inside a customer's world: their data, their tools, their workflows, their security rules. The agent goes live, the engagement winds down, and the FDE moves to the next customer. From then on, every question about that agent ("why did it do this?", "is it still working?", "what is it costing us?") lands on someone who no longer has a terminal open in that environment.
This guide is about closing that gap: what to instrument before you leave, how to keep a view of every deployment, and how to debug a customer's agent without asking for access every time.
Why FDE deployments go dark
- You do not own the environment. The agent runs in the customer's cloud, behind their VPN, sometimes on their laptops. Your access is temporary by design.
- Every customer is different. Different data, different tools, different edge cases. A failure pattern at one customer tells you nothing about the next.
- Failures are quiet. Agents return a plausible answer instead of an error. The customer notices weeks later, and by then nobody remembers what changed.
- The customer needs proof. Before renewal, someone will ask whether the agent actually works, how often it fails and what it costs. Screenshots from launch week do not answer that.
- Data rules are strict. Regulated customers will not let prompts or documents leave their environment without control.
Instrument on day one, not after the first incident
Make observability part of the deployment checklist, like credentials and monitoring for any other service. For each deployment:
- One agent identity per customer and deployment. Name it so it reads in a list:
acme-claims-agent,globex-support-agent. Never mix two customers under one label. - One trace per task. A claim processed, a ticket answered, a report generated. That is the unit the customer thinks in.
- Tags for the business context. The customer's team, region or plan, so you can slice later without redeploying.
- A privacy level the customer signs off on. Decide with them, in writing, what leaves their environment.
- Failures as first-class events. Every tool error, timeout and refused action recorded with its message, not swallowed by a retry.
Agree the privacy level with the customer
Most enterprise conversations stall here, so have the answer ready. Voight offers three levels, set per deployment:
| Level | What leaves the customer's environment | When it fits |
|---|---|---|
| Minimal | Metadata only: tool names, timings, tokens, cost, outcomes | Regulated data, first weeks of trust |
| Standard | Content, with credentials and personal data scrubbed on the machine before sending | Most production deployments |
| Full | Everything as captured | Internal tools, sandboxes, debugging sessions |
On Minimal, you can still see which step failed, how long it took and what it cost; you cannot read the prompt. That is often enough to find the cause, and it is an easy yes for a security team.
Debug without access
When the customer reports a problem, the trace is your remote terminal:
- Find the agent for that deployment and the time window of the report.
- Open the failed traces and read the first failed step, not the last.
- Compare with a healthy run of the same task from the same customer.
- Check what changed: input size, a new document type, a model update, an expired credential on their side.
In Voight, failures are also grouped into issues automatically, so you see "this tool fails with this error, 14 times since Tuesday" instead of 14 separate reports. Diagnose with AI proposes a root cause and a fix from the recorded runs, and the issue tracks whether your fix held: it is marked as verified after five quiet days, or as recurred if it comes back.
Prove the deployment works
Renewals are won with numbers the customer can check. From the same data:
- Runs and success rate per week, per deployment.
- Issues found, fixed and verified, with dates.
- Cost per task, from tokens and model prices, so the customer can compare it with the manual process it replaced.
- Traces of real tasks to walk through in a review meeting, instead of a demo.
A deployment checklist for FDEs
- An agent identity per customer and deployment.
- One trace per task, with business tags.
- Privacy level agreed and recorded.
- Tool errors and timeouts recorded, not swallowed.
- An owner for the issue queue after you leave, on your side or theirs.
- A monthly review using real traces and the numbers above.
Getting started
Coding agents connect with one command, LLM apps with a wrapper around the OpenAI or Anthropic client, and any other runtime through the HTTP API:
npx -y @voightxyz/sdk setup
If you run deployments for several customers and need SSO, dedicated infrastructure or an SLA, that is what the Enterprise plan is for: write to team@voight.xyz.