Give the candidate system two policies. One expired Friday; one replaced it Monday. Ask which policy controls a Tuesday request, then remove access to the current document during the next run. The answer, refusal, and log will tell you more about an LLM development company than a portfolio page can.
The exercise sounds small. It touches source precedence, permission filtering, unsupported-answer behavior, model configuration, evidence retention, and the route to a person. Those are product decisions. They live in the application around the model.
Model choice matters. It is still one line in a larger system: data enters, approved context is selected, a model runs, output is checked, uncertain cases go to a person, and an operator gets a usable record. OpenAI’s API documentation warns that prompting behavior can change between model snapshots and recommends pinned model versions plus application evaluations. A contract tied to a model name will age. A contract tied to a test set can be rerun.
This scope starts after the buyer has decided that a language-model application deserves a build. If that decision is still open, start with the LLM-versus-rules-versus-predictive-ML choice. The work here begins one step later: turning a plausible use case into evidence a team can accept.
Start with the failure sentence
Write one sentence that describes the failure the business refuses to buy.
For an internal policy assistant, it might be: “The assistant cites a retired policy when a current policy exists.” For a contract-extraction tool: “A missing renewal clause is returned as no renewal obligation.” For a support drafting tool: “Customer data from one account appears in another account’s draft.”
These are better starting points than “build a RAG chatbot.” The architecture phrase says nothing about the business consequence. The failure sentence tells a vendor what must be represented in the test data, which source wins during a conflict, who reviews an uncertain result, and what the application should do instead of guessing.
Put the following fields in the scope packet before asking for an estimate:
| Scope field | The decision it must record |
|---|---|
| User and job | Who uses the application and the exact task they are completing |
| Non-goals | Adjacent tasks the application must refuse or route elsewhere |
| Authoritative sources | Which records may answer the task and which source wins a conflict |
| Output contract | Required fields, allowed labels, citation shape, and null behavior |
| Action boundary | Read-only output, proposed action, or an action the system may execute |
| Failure catalog | Real mistakes, harmful near-misses, and unsupported requests |
| Review owner | The person who decides disputed cases and changes the rubric |
| Service budgets | Response-time ceiling, per-task cost ceiling, and outage behavior |
| Evidence period | Logs, samples, and operator feedback retained for acceptance |
The packet does not need to be enormous. It needs to be specific enough that two firms would build toward the same result. If each bidder silently imagines a different product, their estimates are not comparable.
Pick the smallest architecture that can pass
The failure catalog should drive the technical choice.
A prompt and a structured output schema may be enough for a bounded extraction task with all required text in one request. Retrieval earns its place when the answer depends on a larger corpus, current material, source citations, or document-level permissions. Fine-tuning is worth testing when repeated behavior or format remains poor after the simpler baseline is measured. Tool use belongs in the design only when the application needs fresh external data or must call another system.
That last boundary is easy to blur in a demo. Anthropic’s tool-use documentation describes a tool call as a contract between the application and the model: the model emits a structured request, while application code executes the operation and returns the result. The model does not acquire safe authority because a tool description sounds careful. Your code still owns authentication, validation, side effects, retries, and error handling.
Ask the vendor to show a baseline before adding machinery. If the prompt-only version passes, a retrieval stack would add failure modes without earning its keep. If retrieval passes, fine-tuning needs a measured improvement to justify another training and release path. Buyers often receive an architecture diagram before anyone has run a representative case. Reverse that order.
The deeper cost decisions already have their own intent owners: RAG implementation cost and LLM fine-tuning cost. This page is about the evidence required whichever route wins.
Turn acceptance into a dataset
An LLM acceptance test is a set of cases with inputs, permitted context, expected behavior, and a verdict rule. Build it from the work, not from toy prompts invented by the development team.
Include ordinary cases. Then add the uncomfortable material: an answer absent from the source, two documents that disagree, a retired policy beside its replacement, a document the current user cannot access, misspelled entities, an overloaded input, instructions hidden inside retrieved text, an unavailable downstream service, and a request that belongs with a person.
Each case needs a judge. Some outputs can be checked with code: valid JSON, allowed enum, required citation, exact identifier, no secret-shaped text. Other cases need a rubric and a domain reviewer. Keep those judgments narrow. “Good answer” is not a repeatable label. “Uses the current refund window, cites the policy, and makes no promise outside the policy” can be reviewed twice and produce the same argument.
Anthropic’s evaluation guidance says success criteria should be measurable and tied to the actual task. Its test guidance also calls for cases such as missing data, long inputs, harmful inputs, and genuine ambiguity. OpenAI’s Evals API likewise separates the evaluation data schema from testing criteria and graders. Those are vendor examples, but the procurement lesson is provider-neutral: the dataset and rubric are part of the product.
Use a result table that exposes tradeoffs instead of collapsing everything into one average:
| Outcome family | Evidence to retain | Release rule |
|---|---|---|
| Supported answer | Answer, source, retrieved passage, rubric verdict | Meets the business rubric on the approved case set |
| Unsupported answer | Refusal or escalation record | Does not invent a source or action |
| Structured result | Raw output and schema-validation event | Every required field validates or enters a named repair path |
| Restricted source | User identity, source filter, retrieval record | No restricted passage or derived answer appears |
| Tool request | Requested action, validated arguments, authorization result | No execution outside the user’s approved authority |
| Provider failure | Timeout, fallback choice, final user message | Follows the documented degraded-mode policy |
Do not let a high aggregate score hide a catastrophic family. A system that handles routine policy questions well but leaks one restricted document has not passed.
Force a model change before sign-off
Model portability is often sold as a diagram with two provider logos pointing at one box. Test it.
Keep provider-specific calls behind a small adapter. Store the model identifier, prompt version, tool schema, retrieval settings, and output schema as versioned configuration. Then run the acceptance dataset against the proposed primary model and one plausible fallback. Record disagreements by outcome family.
The result will not be identical, and that is fine. Different models may call tools at different points, follow long instructions differently, or return valid structured output at different rates. The useful question is narrower: can the team see the difference, decide which failures are acceptable, and switch without rewriting the rest of the product?
Pin the production model version when the provider supports it. A fallback still needs its own pass rule. An untested fallback is a second production system that happens to be mentioned in a runbook.
This test also reveals vendor lock hidden outside the API call. A proprietary eval dashboard with no export, prompts stored only in a consultant’s account, or retrieval configuration that cannot be recreated is a larger trap than a provider adapter. Put exportable configuration and test records in the handoff list.
Treat context and output as hostile inputs
Retrieved text can contain instructions. So can a web page, uploaded document, support ticket, or tool result. The OWASP GenAI Security Project defines prompt injection as input that changes model behavior in unintended ways and notes that retrieval and fine-tuning do not fully remove that risk.
The control belongs outside the prose prompt. Enforce document permissions before retrieval. Keep secrets out of model context. Validate structured output before code consumes it. Restrict tools to the current user’s authority, and require a separate approval for sensitive or irreversible actions. Log the identity, input class, retrieved record identifiers, model configuration, tool request, authorization result, and final disposition.
Add adversarial cases to the acceptance dataset. A retrieved document can tell the model to ignore prior instructions. A user can ask the assistant to expose its hidden context. A tool result can contain text that looks like a new command. The expected behavior must be visible in test records, not asserted in a security paragraph.
NIST’s Generative AI Profile is a voluntary companion to the AI Risk Management Framework that organizes suggested actions across govern, map, measure, and manage activities. A buyer can turn that lifecycle view into concrete handoff evidence: named owners, documented use and misuse cases, test results, production monitoring, incident response, and a change record.
Own the handoff assets
The application is not handed over when the demo succeeds. It is handed over when another qualified team can operate it without the original consultant’s private notebook.
Require these assets in your repository or approved systems:
- Source code, dependency locks, deployment configuration, and reproducible environment instructions.
- Data-flow and trust-boundary diagrams that name stores, providers, identities, and retention points.
- Versioned prompts, model configuration, tool schemas, retrieval settings, and output schemas.
- The acceptance dataset, rubric, code-based checks, run reports, and known-failure register.
- Monitoring definitions for quality, latency, failures, per-task usage, and escalation volume.
- Operator runbooks for provider outage, bad release, data-source failure, suspected disclosure, and rollback.
- A decision log for rejected architecture options and the evidence behind the chosen one.
- An access inventory that can be reviewed and revoked after the vendor leaves.
No production secret belongs in that package. The handoff should name how secrets are issued and rotated, while the values remain in the approved secret store.
Amazon Bedrock’s evaluation documentation is a useful example of portable evidence: it supports custom prompt datasets, ground-truth responses for retrieval tests, computed metrics, and human review for cases that need judgment. A buyer does not need Bedrock to copy the principle. Keep the cases, expected records, scores, and reviewer decisions in a form the next team can rerun.
Make payment follow evidence
Milestones should end in inspectable work.
The first milestone produces the failure catalog, source inventory, permissions map, initial dataset, and a baseline. The next delivers one thin path through the real environment, including an unsupported case and a human handoff. A later milestone adds the remaining approved cases, operating records, adversarial results, and the tested provider fallback. Final acceptance is the rerunnable handoff, not a presentation.
This structure makes scope changes visible. If the source inventory uncovers six undocumented repositories or the action boundary grows from read-only drafting to production writes, the contract can change against evidence. Nobody has to pretend the original estimate covered a different product.
It also gives the buyer a sane stop point. A baseline that cannot meet the minimum business rubric may show that the use case, data, or model approach is wrong. Stopping there is cheaper than decorating a failed premise with more infrastructure.
Keep agent scope separate
An LLM application can answer, extract, classify, summarize, or draft while a person remains the actor. An AI agent adds a loop in which model output can select tools and change another system. That is a material jump in authority.
If the use case needs actions, add action-level authorization, idempotency, spend and attempt limits, stop conditions, state reconciliation, and approval rules to the packet. Test partial failures: the model asks for an action, the downstream call times out, and a retry could duplicate it. A convincing chat transcript does not cover that branch.
Read the AI agent development company guide for the agent-specific buying decision. If the integration itself is meant to be shared across several AI clients, MCP server development scope owns that narrower choice.
gmware’s AI agents and LLM integration service covers application design, integration, evaluation, and operating handoff. Bring one failure sentence and a handful of ugly real cases to the first scoping call. That is enough to expose a vague proposal quickly.
Next step: Turn your LLM use case into an acceptance packet with gmware