At 2:13 a.m., an AI support agent reports that a refund failed. The payment system shows a completed credit. Your provider has the trace, your application team owns the integration, finance owns the money, and nobody is sure who has authority to stop the agent before it tries again.
That is an observability failure even if every span arrived.
AI agent observability becomes an operating service only when the parties assign detection, access, containment, recovery, change, and exit. Traces tell you what the software recorded. The contract tells a person what to do next, and gives that person enough authority to do it.
We’re gmware, a software development firm headquartered in Austin, Texas, with engineering centers in Bangalore and Mohali, India. We build tool-using agents and the software that surrounds them. Our opinion: write the ownership boundary before anyone promises a four-hour response. A fast acknowledgement from someone who cannot inspect the target system or stop another write is clerical theater.
Our existing AI agent observability guide covers signals, tools, and platform costs. This guide starts where that one stops: who operates the evidence, which events become incidents, and what you still own after hiring a provider.
Follow one run across the ownership seams
Begin with a run that matters. A support agent reads an account, checks policy, calls a refund tool, receives a timeout, and tells the customer the refund failed. The target system accepted the credit before the response disappeared.
Five teams can each own a clean green box while the customer gets two credits. The agent team owns instructions and tool selection. The platform team owns telemetry transport. The application team owns the refund integration. Finance owns correction. A managed provider may watch the alerts. The useful unit of responsibility is the complete business action, not any one console.
Google Cloud’s agent observability documentation includes model interactions, token use, latency, errors, tool calls, and the data exchanged with tools. That is a sensible evidence surface. It still does not choose the person who may freeze refunds, confirm the payment state, or approve recovery. Your operating contract has to add those decisions.
Use a stable run identifier across the agent steps and a stable action identifier for the attempted refund. The second identifier matters when the agent retries. One run can attempt an action twice; one accepted action can outlive the run that started it.
The incident path needs five owners
Write the responsibility record before the SLA
Do not begin with severity labels. Begin with actions, evidence, and authority. Then attach time targets to work that somebody can actually perform.
This compact record is enough to expose most gaps:
| Operating event | Primary owner | Required access | Decision right | Evidence left behind |
|---|---|---|---|---|
| Agent claims success but the target result is unknown | Application on-call | Agent trace plus target-system read | Pause retry and reconcile | Run ID, action ID, target response, resolution |
| Tool request is denied by policy | Agent operations | Denial event and policy version | Escalate repeated or high-impact attempts | Request, denial reason, affected policy, case owner |
| Agent uses an unexpected tool or identity | Security on-call | Tool inventory, identity logs, agent version | Disable the tool or credential | Access record, containment action, review ticket |
| Quality check finds a harmful or materially wrong result | Business owner | Sample, expected result, evaluation record | Remove the version from service | Failed case, decision, replacement version |
| Telemetry stops arriving | Platform on-call | Collector and storage health | Move agent to the written safe mode | Gap window, affected runs, restoration proof |
| Provider needs production trace content | Customer data owner | Time-bounded approved view | Approve, redact, or refuse access | Request, scope, approver, expiry, access log |
Adapt the rows to your system. A read-only research agent and a refund agent should not share the same response model. If the agent cannot change business state, losing a trace may be a diagnostic problem. If it can create a payment, the same gap may require immediate containment.
The table also separates the service provider from the accountable owner. A provider can monitor, investigate, and recommend. Your organization should still name who accepts operational risk, who authorizes a return to service, and who resolves the business consequence.
Make the SLA describe agent failure
An API can return 200 while an agent chooses the wrong account, calls the wrong tool, or reports success before the target accepts the write. Availability and latency still matter. They just aren’t the whole service.
Define the covered agents, versions, environments, hours, and tool boundaries first. Then write the event classes. Useful classes include an unauthorized request, an accepted action with an unknown result, repeated tool loops, missing required approval, cross-customer access, a false completion message, and loss of telemetry for a state-changing path.
Each class needs its own clock. Detection time measures when the monitoring system first recognized the condition. Acknowledgement time measures when the assigned operator accepted the case. Containment time ends when further harm is bounded. Recovery time ends when the business state is corrected or moved to an approved manual path. These clocks answer different questions; collapsing them into “response time” makes a clean report and a bad incident.
Avoid a universal threshold copied from another workload. Write the target from the possible consequence, operating hours, and available stop mechanism. A wrong internal draft can wait for business hours. A tool that can send money or disclose another customer’s record needs a different rule.
Treat trace access as production access
Agent traces can carry much more than timing data. The OpenTelemetry GenAI attribute reference warns that input messages and output messages are likely to contain sensitive information. Retrieval queries can also contain sensitive material. Tool arguments may include the exact record an agent tried to read or change.
Write a telemetry data rule with the same care you give the application database. Specify content captured by default, fields redacted before export, sampling rules, encryption, retention, storage region where relevant, approved roles, emergency access, and deletion. Do not give every dashboard user raw prompt and response access because it makes debugging convenient.
Record the provider’s subprocessors and the path data takes between the agent, collector, processor, and storage. Keep customer-controlled access for the evidence needed to investigate a disputed action. If all raw traces live behind the provider’s account, your team may lose the record at the same moment the commercial relationship breaks.
Put enforcement outside the prompt
Observability finds behavior. It does not make broad authority safe.
OWASP’s guidance on excessive agency recommends limiting tool functionality and permissions, executing actions with the appropriate user context, and enforcing authorization in downstream systems instead of asking the model to decide whether an action is allowed. The observability contract should prove those controls were called and record their decisions. It should not substitute a watchful dashboard for them.
For a refund action, log the requesting identity, policy version, normalized action, authorization result, target response, and retry decision. Do not put secrets into the trace. Do not treat the model’s explanation as proof of authorization. The downstream decision and target state are the stubborn facts.
This is also where AI agent evaluation and production observability meet. Evaluation challenges a version before release. Observability supplies production-shaped failures for the next evaluation set. A serious incident should change at least one of the test suite, permission boundary, alert rule, recovery path, or deployment decision. If it changes none of them, the review produced minutes rather than learning.
Drill the ambiguous write
The best contract test is not a clean model error. Break a dependency after it accepts an action but before the agent receives the response.
Run the drill in a safe environment. Confirm that the operator can correlate the run and action, read the destination state, stop an automatic retry, reach the business owner, correct the result, and preserve a review packet. Then swap the on-call person. A process known only to the engineer who built it is an anecdote, not an operating model.
The drill will uncover dull failures. The alert points to a dashboard the provider cannot access. The customer on-call can see the trace but not the payment record. The stop switch disables new conversations but leaves queued actions running. The escalation contact left the company three months ago. Good. These are cheap findings in a drill.
Our AI agent audit guide explains how to reconstruct one material action after the event. The observability agreement has a more immediate job: make the evidence available while someone still has time to contain the next action.
Negotiate the exit while everyone is friendly
An exit clause that says “customer data will be returned” is too thin. Name the artifacts and formats.
Require telemetry schemas, collection configuration, redaction rules, alert definitions, dashboards, evaluation definitions, incident records, change history, runbooks, access inventory, data locations, retention settings, and known blind spots. Decide which items transfer as editable configuration and which arrive as exports. Test one export before renewal, not during termination week.
Assign the last-day work. Revoke provider identities. Rotate shared credentials. Transfer alert destinations. Confirm that customer-owned collectors still send data. Move open incidents and exceptions to named owners. Obtain the agreed return or deletion evidence for retained content.
There is an honest non-fit here. If your team runs one low-volume, read-only agent and can inspect every failed run, a managed observability service may be more ceremony than help. Keep structured traces, a failure queue, a named reviewer, and a tested disable path. Buy the operating layer when volume, consequence, coverage hours, or team boundaries make that simple setup unreliable.
Bring one ugly incident to discovery
GM Ware builds AI agents with the telemetry, permission checks, evaluation gates, and recovery paths needed to operate them. Our AI Agents & LLM Integration service can include an observability operating model, but we won’t pretend a dashboard transfers accountability.
Reach out with one failure that worries you: a duplicate action, a silent wrong answer, a missing approval, or a trace your provider can see but you cannot. We’ll map the evidence path, owner, stop right, and exit artifact before recommending a platform.