AI Agent Observability

Why ordinary monitoring misses it

Traditional observability was built for systems that fail loudly. A service is up or down, a request returns 200 or 500, latency is inside the budget or outside it.

Agents fail quietly. Every call succeeds. Latency is normal. Error rate is zero. And the agent is confidently doing the wrong thing, because a model changed, a prompt was edited, a tool started returning a slightly different shape, or the input distribution drifted away from what it was tested on.

Nothing in your dashboard turns red. The first signal is a customer complaint, or a number in a report that does not make sense.

What to track

Execution traces

The full path of every run: the input, each step taken, each tool called and what it returned, the model calls with their prompts and responses, and the final output. Not sampled. When someone asks what the agent did in a specific case, sampling means you cannot answer.

Cost per execution

Tokens in, tokens out, by model, by step, by agent. Aggregated per day and per customer where that applies. Cost is the metric that moves fastest and surprises hardest, and it is usually the one nobody instrumented until after the invoice.

Latency, broken down

Total duration is not enough. You need to see which step is slow. Usually it is one tool call or one over-large model invocation, and you cannot fix what you cannot locate.

Failure modes, separated

Agents fail in categories that need different responses: the model refused, a tool errored, the output failed validation, the agent hit its step limit, it looped. Lumping these into one error rate destroys the information.

Quality

The hard one. Options that work: validate output against a schema and track the failure rate, run a sample against a reference set, use a model to grade output against criteria, and track how often a human overrides or corrects the agent. That last one is the most honest signal you will get and the cheapest to collect.

Version attribution

Every execution tagged with the agent version, prompt version and model that produced it. Without this you cannot answer whether last Tuesday’s change caused this week’s problem, which is the question you will actually be asked.

Observability and audit are not the same

Observability is for your engineers, so they can debug. Audit is for everyone else, and it has different requirements.

An audit trail has to be immutable, complete rather than sampled, retained for a defined period, and attributable to a person. It answers who authorised this agent to run, which version was live, what it did, and what data it touched.

If you operate in banking, healthcare, insurance or the public sector, this is not a nice-to-have that you add later. Under the EU AI Act, systems classified as high-risk carry logging and record-keeping obligations, and retrofitting them is considerably harder than building with them.

Instrument before launch

The standard order of events is: launch, discover a problem, add logging to diagnose it, wait for it to happen again. That is two incidents instead of one, and the second one is the expensive kind.

Before an agent handles real work, have traces recorded, cost alerts set, failure alerts split by category, and version tagging on every execution. It is a day of work in advance and it saves considerably more than that. The rest of the production checklist.

Common questions

Can I use my existing APM tool?

Partly. It handles latency and errors. It does not handle token cost, prompt versioning, tool call traces or output quality, because it was not built for non-deterministic systems.

Should I log prompts and responses?

For debugging, yes. For privacy, carefully. If your platform runs in your own infrastructure, the logs stay in your perimeter and this is a much simpler conversation.

How long should I retain traces?

Long enough for the regulation that applies to you. For regulated verticals plan in years, not weeks, and price the storage in from the start.

What is the single most useful metric?

The human override rate. How often a person corrects or reverses what the agent did. It is cheap to capture and it tracks real quality better than anything else on this list.

Stackbone

Stackbone records every execution with full traces, cost, latency and version attribution, and keeps an immutable audit trail of which agent version ran on whose authority. Everything runs in your own cloud account, so the trace data never leaves your perimeter.