The useful first meeting with an AI automation agency should be uncomfortable. Give the team a real exception, not the clean sample that makes the demo sing. Ask what the system does when a purchase order has two totals, the customer record has no stable ID, or the approval API times out after accepting a request.
A good agency slows down at that point. It asks which source is authoritative, who may decide, what evidence the reviewer needs, and whether the action can be reversed. A weak one changes the subject to model choice.
We’re gmware, a software development firm headquartered in Austin, Texas, with engineering centers in Bangalore and Mohali, India. We build through our AI and machine learning practice. We think the audition matters more than the portfolio because your failure path will never appear in somebody else’s case study.
This guide is for the buying decision. Our AI automation services guide covers which workflow to automate first. Here, the question is narrower: which agency can turn that workflow into a system you can test, operate, and eventually change without calling the original demo team every time?
Run the selection as a five-day audition. Each day should use evidence produced on the day before it. Agencies do not need to build a production system in a week. They do need to show how they think when the clean path ends.
Day zero: send the same failure packet
Choose one bounded slice of the proposed workflow. Supply representative inputs, including the awkward cases operators already know. State the allowed action, forbidden action, review point, and evidence that would count as a correct result.
For an invoice-intake project, the proof task might read like this:
Extract supplier, invoice number, currency, total, and purchase-order reference from 40 supplied invoices. Flag duplicates. Do not create or approve a payment. Route unreadable documents, conflicting totals, and unmatched suppliers to an accounts-payable reviewer with the source attached.
That brief is small enough to test and serious enough to expose judgment. Every candidate gets the same packet. Letting agencies choose their own demonstration is like letting job candidates write the exam after seeing the answers.
The proof task also prevents category drift. Optical character recognition, a deterministic rule, a workflow engine, and a language model may all belong in the answer. The buyer should not pay for AI where a field mapping will do.
Day one: make the agency narrate failure
Most sales demos define success as a correct final output. Production systems need more states.
- Success: the input is within scope and the result satisfies the written rule.
- Abstention: the system detects that evidence is missing or ambiguous and sends the case to the correct person.
- Controlled failure: a dependency is unavailable, so no side effect occurs and the case remains recoverable.
- Unsafe result: the system attempts an excluded action, exposes restricted data, or creates a result that cannot be traced.
Ask the team to put a number on none of these unless it has a repeatable evaluation behind it. A percentage without the test population, scoring rule, and sample failures is decoration. Save the four definitions; they become the expected outcomes for day two.
The NIST AI Risk Management Framework organizes AI risk work around Govern, Map, Measure, and Manage. It is voluntary and use-case agnostic, which makes it useful as a diligence frame rather than a certificate. Ask an agency what it maps, how it measures, who governs changes, and what evidence causes it to manage the system differently.
Day two: turn the failure into a test
A slick interface can sit on top of a brittle decision. Spend the technical review on the evaluation set.
Look for real variation: incomplete records, old templates, duplicate events, conflicting sources, adversarial text, late responses, changed schemas, and permission errors. Each case needs an expected outcome and a reason. The agency should preserve failed cases and add them to regression tests after a fix.
Separate component tests from workflow tests. Correct extraction does not prove that the right customer record was updated. A working API call does not prove that a retry cannot create the same order twice. The end-to-end test must follow the business event through every system it can affect.
Ask who can change the test set. If the same person can weaken acceptance criteria to make a release pass, the evaluation is not an independent gate. Pick one failed case for the authority review on day three.
Day three: remove authority and run it again
“Human in the loop” is too vague for a contract. Name the decision and the person.
A useful authority ladder starts with read-only observation. The next stage may allow the system to prepare a draft. Later, a person approves a bounded action. Direct execution belongs at the top and should require evidence that matches the possible harm.
For each stage, record:
- the identity used by the automation;
- the systems and records it may read;
- every action it may request or execute;
- the value, volume, or time limits on that action;
- the reviewer who can approve an exception; and
- the switch that stops further actions.
The agency should be comfortable with smaller permissions. Resistance here usually means the proposed design depends on convenience that the buyer will carry as risk. Keep the reduced credential for the dependency test rather than returning to an administrator account.
Day four: break one dependency
The model is only one component. Most operational failures happen around it: expired credentials, changed fields, rate limits, out-of-order events, duplicate delivery, and a write that succeeds while the caller sees a timeout.
Give the candidate one of those failures during the proof task. Ask how the system knows whether to retry. A safe answer uses stable event identifiers, reads the destination state where possible, and reconciles uncertain outcomes. “Retry three times” is not enough when the action charges a card, sends a message, or creates a record.
Review the agency’s integration work with the same care as its prompts. The OWASP API Security Top 10 calls out broken authorization, unrestricted resource consumption, improper inventory management, and unsafe consumption of APIs. Those are practical procurement questions: which identity can call what, what is capped, where are all endpoints recorded, and how are third-party responses treated before they enter your workflow?
The failure trace, retry decision, and recovery evidence become the handoff packet for day five.
Day five: put your operator in the chair
Production ownership starts when the demo team leaves. The proposal should name the dashboards, alerts, logs, evaluation schedule, incident route, change approval, and cost controls that remain.
Ask for one operating review in advance. What will the team inspect weekly? Which alert wakes somebody up? What evidence would pause the automation? Who owns a vendor deprecation notice? How is a failed case replayed after repair?
This is also where recurring cost becomes legible. Model calls, workflow runs, observability, data storage, support, and human review are different lines. Ask the agency to expose each one. A low build quote with an unbounded review queue is not low cost.
Rehearse the exit before awarding the work
Handoff is a deliverable, not a farewell call. The exit package should include:
- source code and deployment definitions;
- current prompts, rules, schemas, and model settings;
- the evaluation set with expected results;
- an inventory of accounts, credentials, vendors, and data stores;
- architecture and data-flow diagrams;
- dashboards, alerts, runbooks, and incident history; and
- a recorded restore or redeploy exercise run by the receiving team.
Ownership of those artifacts must be explicit. So must the right to export operational data and evaluation history. If the buyer cannot reproduce the system in an account it controls, the agency has sold dependency along with automation.
Score the evidence after the audition
Use a zero when an agency offered only a promise, a one when it described an approach, and a two when it produced the artifact or demonstrated the behavior. Scoring last matters. It stops a confident presentation from earning points before the team has handled a failure.
| Test | Evidence worth two points | Automatic concern |
|---|---|---|
| Problem boundary | Written inputs, outputs, excluded actions, and owner | ”We will discover that after kickoff” |
| Evaluation | Versioned cases with pass, fail, and abstain outcomes | Accuracy claim with no test set |
| Failure behavior | Timeout, duplicate, ambiguity, and partial-write demonstrations | Happy path only |
| Data access | Named sources, retention, access scope, and deletion path | Broad production copy for convenience |
| Tool authority | Read, draft, approve, and execute permissions separated | One credential can do everything |
| Integration | Identity, retries, idempotency, reconciliation, and rate limits | Connector list presented as architecture |
| Operations | Logs, alerts, triage owner, cost limits, and recovery drill | ”The model learns over time” |
| Handoff | Repository, deployment record, runbook, test set, and credential transfer | Continued operation depends on one person |
Do not turn the total into false precision. A firm with fourteen points can still be wrong for the job. The scorecard makes missing evidence visible so the decision meeting can deal with it.
If you only have 30 minutes
Use the first ten minutes on the proof-task boundary. Spend the next ten on one failed example and the resulting control. Use the last ten on operation and exit.
Skip the generic capability deck. Ask these instead:
- Which part of this task should not use AI?
- Show one case where the system must abstain.
- What is the most damaging plausible side effect?
- Which credential performs the action, and how is its scope limited?
- How would we know that a source schema changed?
- What will our team be able to run without you on day one after handoff?
The answers will not all be perfect in a first call. They should be concrete. The agency should ask for facts it needs rather than filling gaps with sales language.
Our recommendation
Shortlist on relevant delivery experience, but select on the proof task. The best AI automation agency for your project is the one that makes the boundary sharper, shows you how the system fails, and leaves behind evidence your own team can use.
If every candidate passes the demo, make the demo harder. Production already will.
If you want an engineering team to scope the proof task before a wider build, bring one real workflow and its messiest exception to a gmware discovery session.