A support agent passes 18 of 20 test cases. The two misses are a duplicate refund and an account lookup outside the requesting user’s scope.
That is not a 90% release. It is a stopped release.
AI agent evaluation gets distorted when every case contributes equally to one average. A typo in a friendly closing line and an unauthorized write cannot carry the same weight. The evaluation needs to preserve failure severity, prove the state of external systems, and show which exact agent configuration ran. Otherwise, the score is theater with decimals.
This guide turns an eval suite into a release gate. The examples use a hypothetical support agent because refunds, account boundaries, approvals, and recovery are easy to inspect. Replace them with the real states and authorities in your workflow.
Download the AI agent evaluation release-gate template. It is a starting record, not a universal scoring standard.
Start with the business state
An agent’s final message is only one output. If it says “refund completed” while the payment system rejected the request, the prose can look perfect and the job still failed. Begin each case with a business state that an independent check can inspect.
For the support example, the state might include the authenticated customer, order status, prior refunds, permitted amount, approval requirement, and idempotency token. The case then names the allowed action and the state expected afterward. A database query or API read verifies that state without trusting the agent’s narration.
Anthropic’s agent-evaluation guidance distinguishes the transcript from the outcome. A transcript records the trial’s steps. The outcome is the final state of the environment. That distinction should appear in the acceptance record.
| Evidence layer | Direct question | Preferred check |
|---|---|---|
| Starting state | Did the test begin with the intended customer, order, permissions, and policy version? | Fixture manifest plus identity check |
| Tool behavior | Did the agent request an allowed tool with valid arguments? | Structured trace assertion |
| Enforcement | Did code outside the model allow or deny the action correctly? | Policy-engine or API response |
| Business outcome | Does the target system contain the expected final state? | Direct read from the test system |
| User result | Did the message describe the actual outcome without hiding uncertainty? | Rule check plus calibrated review |
| Recovery | Can the operator reconcile or reverse the test side effect? | Observed recovery drill |
Do not use a model grader for a fact the target system can answer. “Was one refund created?” belongs in code. “Did the explanation tell the customer enough without sounding accusatory?” may need a rubric and human calibration.
Freeze the system under test
The phrase “we evaluated the agent” hides too much. Record the model identifier, prompt version, tool schemas, permission policy, retrieval snapshot, memory state, harness version, grader version, retry rules, and resource budget. Change one of those and you have changed the tested system.
OpenAI’s third-party evaluation guidance makes the same validity point: the harness, tools, context handling, retry behavior, scoring, and budget can alter the observed result. A release report should identify those conditions rather than attaching a score to a model name alone.
Use a compact run manifest:
| Manifest field | Example entry |
|---|---|
| Release candidate | support-agent/2026-09-25.3 |
| Model and settings | Provider model ID plus pinned reasoning and sampling settings |
| Instructions | Commit or immutable prompt version |
| Tools | Schema versions and test endpoints |
| Authority | Test identity, scopes, approval policy, and spend ceiling |
| Data | Fixture set, retrieval snapshot, and reset version |
| Harness | Runner commit, timeout, retries, and trial count |
| Graders | Code checks, rubric versions, and calibration record |
This manifest also prevents a common argument after a failed run: “That is not the configuration we meant to test.” The record says what ran.
Build cases from costly boundaries
The happy path belongs in the suite, but it should not dominate it. Start with the decisions that can hurt a customer or create cleanup work. For a tool-using agent, those boundaries usually sit where identity, money, external communication, or irreversible state changes meet uncertain input.
The OWASP excessive-agency guidance connects harmful agent actions to excessive functionality, permissions, or autonomy. It recommends limiting tools and permissions and enforcing authorization in downstream systems. Turn those controls into executable cases.
| Case | Test pressure | Passing evidence | Severity if failed |
|---|---|---|---|
| Authorized refund | Eligible order inside the user’s account | One refund, correct amount, matching receipt | High |
| Duplicate request | Same event arrives twice | One business action and a recorded duplicate | High |
| Cross-account request | Prompt names another customer’s order | Denial before protected data or action is exposed | Release blocker |
| Approval boundary | Amount exceeds the automatic limit | No refund until the named approver acts | Release blocker |
| Tool timeout | Payment request times out after submission | Reconciliation before any retry | High |
| Prompt injection | Retrieved text asks the agent to ignore policy | Instruction treated as data; no added authority | Release blocker |
| Missing context | Order ownership cannot be established | Clear stop and review route | Medium |
| Recovery | Earlier test leaves an uncertain state | Operator reaches a reconciled state from the runbook | High |
Every row needs a reference result that a domain owner accepts before the agent runs. If two qualified reviewers cannot agree on the expected decision, repair the case. Do not ask a grader to resolve a policy dispute that the business has not settled.
OpenAI’s business guidance on evals recommends a test environment that mirrors real conditions, examples drawn from real work, and rare cases that are costly when mishandled. It also keeps domain experts involved in rubric design and grader review. That is a sound operating split: software checks the facts it can prove; accountable people define the judgment calls.
Run more than one trial when variation matters
One pass proves one path through one run. An agent may choose a different tool order, retrieve a different document, or word an ambiguous approval request differently on the next attempt. Repeat trials when that variation could change a material outcome.
Do not choose a trial count by habit. Choose it from the decision. A deterministic denial enforced by an API may need fewer agent trials than a long conversation with several tool choices. A rare but severe failure may demand a larger test budget or a direct control that makes the dangerous action impossible.
Report the denominator. “No failures observed” is incomplete without the number of trials, the case mix, and the configuration. Keep each attempt separate so one clean rerun cannot erase the failed trace.
Anthropic defines a task as one test case and a trial as one attempt, then recommends multiple trials because model outputs vary. Its guidance also separates capability evals, which probe what the agent may be able to do, from regression evals, which protect behavior already expected to work. A release gate needs the second set even while the team is exploring the first.
Grade in layers
A useful grader stack has a pecking order.
First, check hard facts with code: target state, record count, amount, identity, policy decision, required approval, duplicate handling, latency ceiling, and spend ceiling. These checks should fail loudly.
Next, apply a narrow rubric where code cannot decide the quality. Define each rating with positive and negative examples. Ask whether the response disclosed an uncertain outcome, followed the approved tone, or requested missing information without inventing it. Keep the rubric attached to the business purpose.
Then sample the traces. A high aggregate score can hide reward hacking, a broken case, or an agent that reaches the expected state through an unsafe path. OpenAI’s evaluation-validity guidance calls out shortcuts, contamination, refusals, broken problems, and harness effects as reasons a score can misstate the intended capability. A reviewer should inspect samples for exactly those problems.
Last, preserve human override as a recorded decision. The owner may reject a passing run because a new failure mode sits outside the current graders. That is not a defect in automation. It is evidence that the suite needs another case.
Turn severity into a release decision
Write the gate before results arrive. A practical policy can combine blockers with score floors:
- Zero unauthorized actions, cross-customer disclosures, bypassed approvals, or unreconciled side effects.
- Every high-severity case passes its required number of trials.
- Normal cases meet their declared outcome and user-result floors.
- Latency and cost stay within the tested budget for the case mix.
- Any manual override names the owner, reason, expiry, and follow-up case.
The numbers in your policy come from the business risk, not this article. A company might permit a wording defect and forbid a single duplicate payment. Another might require human approval for every external message during its first rollout. Both can be defensible if the authority and evidence are explicit.
Avoid a weighted total that lets twenty easy cases cancel one dangerous miss. Keep release blockers visible as blockers. The arithmetic should make the decision easier to inspect, not wash it clean.
Keep the suite honest after launch
An eval suite ages. Tool responses change, policies move, user behavior finds new corners, and a once-difficult case may become trivial. The suite needs an owner and a change record.
The NIST AI RMF Measure playbook calls for documentation of test sets, metrics, tools, and test, evaluation, validation, and verification methods. It also connects safety measurement with reliability, robustness, monitoring, and response to failures. Treat that documentation as operating material, not a one-time appendix.
After launch, route confirmed failures back into the case bank. Sample production traces only under the organization’s data-access, privacy, and retention rules. Re-run the regression suite when the model, prompt, retrieval source, tool, permission, workflow, grader, or fallback changes.
Evaluation and observability are adjacent, not interchangeable. The AI agent observability guide covers production signals and operating cost. The AI agent audit guide covers reconstruction of a material action. Evaluation asks a narrower pre-release question: did this version pass the agreed cases under the recorded test conditions?
The procurement test
When a supplier presents an AI agent evaluation, ask for one failed case, one raw trace, one direct outcome check, and the release policy. Ask who wrote the expected results and who can stop deployment. Then ask the supplier to rerun a case after changing a tool permission or simulating a timeout.
You are testing the evaluation system as much as the agent. A dashboard screenshot is weak evidence if no one can connect its score to the target-system state, the tested configuration, or a release decision.
GM Ware builds and integrates tool-using agents with test harnesses, permission boundaries, recovery paths, and operating evidence. Bring a real workflow, two ugly exceptions, and the person who owns the final business state to a working session about AI Agents & LLM Integration. The first useful result is often a release gate that rejects the current demo for a precise reason.