I do not trust an agent deployment until I can replay what it tried to do. The model choice matters, but the operational question is simpler: when a run fails, gets expensive, or does something surprising, can the team see the workflow, tool calls, model latency, token use, and sensitive data boundaries without reading raw prompts in a panic?
That is why my default for production AI systems is not "add observability later." It is a small telemetry contract before the first serious rollout. Recent docs from OpenTelemetry, OpenAI, Claude Code, and AI SDK 7 all point in the same direction: agent systems are becoming ordinary distributed systems with weird payloads, and they need boring traces.
The contract I want first
A useful agent trace does not need to capture everything. It needs to answer five questions consistently:
| Question | Signal I expect |
|---|---|
| What workflow ran? | stable workflow name, trace id, request or conversation id |
| Which model calls happened? | provider, model, latency, input tokens, output tokens, reasoning tokens when available |
| Which tools ran? | tool name, approval wait, execution time, result status, error type |
| What did it cost? | token counts, cache reads, retries, rate limits, final usage |
| What was intentionally hidden? | explicit controls for prompts, outputs, runtime context, and tool context |
The lazy version is a naming contract plus a few required fields. Do not start by building a custom dashboard. Start by making sure every agent run emits the same trace shape, then let Datadog, Honeycomb, Grafana, Langfuse, Sentry, or whichever backend is already paid for do its job.
Why OpenTelemetry is the right default
OpenTelemetry's GenAI semantic conventions now cover the pieces that used to force every team into custom attribute names: agents, workflows, tool calls, retrieval, usage, provider names, evaluation events, and reasoning token counts. The important part is not that every field is stable forever. The important part is that the industry is converging on shared names for the same operational facts.
That changes the build decision. If I name my spans and attributes around OpenTelemetry conventions, I can swap collectors and backends later. If I invent agentStuff.modelMillis and tool_json_blob, I am signing up for migration work the first time a vendor ships better native GenAI support.
The current conventions also make a useful distinction between spans, events, and attributes. Tool execution with duration belongs in a span. A retry, exception, evaluation score, or approval decision can be an event. Request metadata that is useful for filtering belongs in attributes. Raw prompts and tool results should be opt-in, because they are usually where secrets, customer data, and embarrassing internal context leak.
SDKs are starting to agree
OpenAI's Agents SDK traces workflows by default and breaks the run into spans for the agent, model generation, function tools, guardrails, handoffs, and audio operations. It also calls out the privacy trap directly: generation spans and function spans may store inputs and outputs, and teams can disable that capture.
Claude Code's agent observability docs describe a similar operational shape through OpenTelemetry export. The useful details are practical: there are spans for model requests, tool execution, permission waits, hooks, and subagent delegation. That is the shape I want when debugging a production run, especially when a human approval step or a slow tool made the agent look like it was thinking.
AI SDK 7 moves in the same direction for TypeScript teams. The release adds a dedicated @ai-sdk/otel package, GenAI semantic convention spans and metrics, lifecycle callbacks, step performance statistics, Node.js tracing channel support, and explicit filters for runtime and tool context. The API detail matters less than the product direction: telemetry is no longer a per-call afterthought. It is becoming part of the agent runtime.
The minimum instrumentation policy
For a production agent, I would make these rules boring and mandatory:
- Every run has a workflow name that maps to a real product capability, not a component name.
- Every run has a stable group id, usually the request id, conversation id, issue id, or job id.
- Every model call records provider, model, latency, status, token usage, and retry count.
- Every tool call records tool name, approval state, execution time, status, and error type.
- Prompt, output, runtime context, and tool context capture are disabled by default unless the field is deliberately allowed.
- Evaluation results are emitted as events, not buried in logs.
- Cost and latency are visible per workflow, not only per provider account.
That is enough to operate the first version. It also leaves room for stricter governance later, like the allowlists and approval boundaries I wrote about in remote MCP server governance. Observability does not replace those controls. It gives you evidence when they fire.
A tiny TypeScript shape
I would rather standardize a small object than argue about dashboards in every feature review:
export type AgentTelemetryContext = {
workflowName: string;
groupId: string;
userVisibleFeature: string;
captureInputs: false;
captureOutputs: false;
};
export const telemetryContext = (input: {
workflowName: string;
groupId: string;
userVisibleFeature: string;
}): AgentTelemetryContext => ({
...input,
captureInputs: false,
captureOutputs: false,
});
The important decision is the default. Product metadata can flow into traces. Raw user data and tool payloads stay out until a team deliberately opts in for a narrow reason.
What I would review in a PR
When someone adds an agent workflow, I would scan for three failures.
First, no stable grouping. If traces cannot be joined back to a customer request, support ticket, GitHub issue, or background job, the trace will be interesting but not actionable.
Second, tool calls hidden inside one giant generation span. Agents are slow or risky because they act. Tool boundaries need their own timing, status, and approval data.
Third, prompt capture turned on casually. This is the same review instinct as treating agent instruction files like production code: the hidden control plane deserves review because small text changes can become large behavior changes.
The forward edge
The next useful improvement is not prettier traces. It is connecting traces to evals, release gates, and incident review. If a new agent version raises tool retries, cost per completed workflow, or approval wait time, that should be visible before customers explain it for you.
The agent stack is moving fast, but this part can stay simple. Give every run a name, a group id, visible model and tool spans, explicit cost signals, and conservative privacy defaults. That is enough structure to debug the system you actually shipped.