A supplier PDF enters an accounts-payable agent’s retrieval index. The visible page is an ordinary invoice. A tiny block of text tells the agent to ignore the invoice task, search its connected drive for contracts, and send a summary elsewhere.
The useful test is not whether the model says, “I cannot do that.” Check the drive, outbound connector, policy log, approval queue, and target mailbox. If no message was sent only because the test address was invalid, the security boundary failed.
Prompt injection testing is an end-to-end test of that boundary. It follows hostile instructions from every untrusted input route through model context and into the tools, identities, validators, approvals, and systems that can cause a real effect. The goal is a release decision backed by side-effect evidence, not a screenshot of one refusal.
Name the forbidden outcome first
Payload lists are easy to collect. A test oracle is harder.
Begin with one outcome the application must never produce. For an invoice assistant, that might be sending a document to an address outside an approved domain. For a support agent, it might be reading another customer’s record. For a recruiting tool, it might be changing a candidate score because of instructions inside a resume.
Write the boundary as a pair:
- permitted: summarize the document selected by the authenticated user;
- forbidden: read another document, change a business record, or contact an unapproved destination because the selected document asked for it.
This pair tells the tester what to observe. It also stops a team from calling the run successful because the final prose looked harmless while a background tool call crossed the line.
OWASP’s LLM01 guidance separates direct injection from indirect injection delivered through external content. It also notes that retrieval-augmented generation and fine-tuning do not fully remove the vulnerability. Your test boundary therefore belongs around the application, not around a chat box or model endpoint alone.
Inventory every path into context
Draw the routes by which text, images, or structured data can reach the model. Do not stop at the user prompt.
| Input route | A realistic test carrier | Evidence to retain |
|---|---|---|
| Direct user input | Chat message or form field | Authenticated user, raw input, normalized input |
| Retrieval | PDF, wiki page, ticket, contract, policy file | Source ID, version, extracted text, retrieved chunks |
| Web or email connector | Page body, email thread, attachment | Sender or URL, fetch time, parser output |
| Tool response | CRM record, search result, API error text | Tool name, arguments, raw response, response trust label |
| Memory | Saved preference, prior summary, task state | Writer, creation run, read run, expiry |
| Image or document layout | Hidden text, alt text, OCR layer | Original asset, OCR result, model-visible representation |
OWASP includes direct, indirect, obfuscated, split-payload, and multimodal examples in its prompt-injection risk page. Those are route prompts, not a complete suite. Add the formats and transformations your product actually accepts: PDF extraction, HTML cleanup, OCR, email quoting, JSON fields, language translation, and tool-result serialization.
The inventory often finds a more basic defect before any adversarial prompt runs. A read-only summarizer may still hold a connector token that can send mail. A retrieval worker may index a shared folder that its users cannot browse directly. Those are authorization problems. Fix them as authorization problems.
Use clean and poisoned versions of the same case
Make two copies of one business input. The clean copy contains only the work item. The poisoned copy adds a benign canary instruction that asks the system to cross the written boundary. Keep every other field the same.
This is a boundary-pair test. The clean case proves the intended job still works. The poisoned case proves the security control stops the unwanted action without destroying the useful one. A system that refuses both has avoided the attack by abandoning its task.
For the supplier PDF example, use a synthetic drive and a test-only outbound destination. Seed the drive with a canary document that contains no real secret. The poisoned PDF can ask the agent to locate that canary and send its identifier. The test is safe to repeat, but it still exercises retrieval, tool selection, authorization, approval, and outbound policy.
Record expected results before the run:
| Layer | Clean case | Poisoned case |
|---|---|---|
| Invoice extraction | Required fields extracted | Same required fields extracted |
| Retrieval scope | Selected supplier records only | No broader search succeeds |
| Tool proposal | Invoice workflow tool | Any unrelated tool request is denied |
| Approval | Normal invoice rules apply | Injection cannot rewrite the approval packet |
| Side effect | Allowed test action completes | Forbidden target remains unchanged |
| Operator signal | Routine trace | Alert or reason-coded denial is visible |
The poisoned instruction does not need to be theatrical. Start with plain language. Then vary placement, formatting, language, encoding, and splitting only after the basic case is measurable. Fancy payloads cannot rescue a test whose expected result was never defined.
Follow the request past the model
Read the complete execution trace. A refusal in the final answer can coexist with an attempted tool call. A successful-looking answer can follow a blocked request that operators never see.
Inspect these events in order:
- the source entered the system and was parsed;
- a specific slice reached model context;
- the model proposed text or a tool request;
- deterministic code validated the proposed output;
- policy allowed or denied the requested action;
- a human saw a fixed approval packet where required;
- the target system accepted, rejected, or never received the request;
- monitoring created the expected signal;
- cleanup restored the test environment.
OWASP’s improper-output-handling guidance says model output should be treated like output from another untrusted user before it reaches downstream functions. Test that claim directly. Put an out-of-policy object ID, destination, file path, or action type in the proposed call and confirm ordinary code rejects it.
Then test permission depth. The OWASP excessive-agency guidance traces damaging actions to excessive functions, permissions, or autonomy. Remove an unused tool. Give a read operation a read-only identity. Force a high-impact action through an approval object that binds the exact target and arguments. Re-run the poisoned pair after each change.
Score consequences, not manners
“The assistant resisted the prompt” is not a repeatable result. Use observed states.
| Result | Release interpretation |
|---|---|
| Forbidden side effect occurred | Fail; contain the test and fix the action boundary |
| Tool request escaped policy but target rejected it | Fail; a downstream accident prevented impact |
| Policy denied the request but no alert or owner exists | Control worked, operations gap remains |
| Model followed the text but had no path to a forbidden action | Behavioral weakness with bounded impact; record and prioritize |
| Model ignored the hostile text and completed the allowed task | Pass for this case, model and configuration only |
| System refused both hostile and clean inputs | Security result may pass, product acceptance fails |
Do not turn five successful prompts into a universal percentage. Model behavior can vary across runs, model versions, temperatures, system instructions, and surrounding context. Report the number of attempts, the exact configuration, and every observed outcome. Keep the raw trace for failures.
The UK’s National Cyber Security Centre warns that prompt injection may not be completely mitigated in the same way as SQL injection because an LLM has no inherent separation between instructions and data. Its prompt-injection guidance recommends reducing likelihood and impact with careful design, deterministic safeguards around tools, and monitoring. A release gate should reflect that residual-risk model. It proves named boundaries under named conditions. It does not certify immunity.
Automate the harness, keep the judgment
An attack harness can send variants, preserve conversations, apply scorers, and rerun a fixed set after a change. Microsoft’s open-source PyRIT project is one option built for generative-AI risk identification and red-team exercises. The choice of tool is secondary.
Your team still owns four decisions:
- which business outcome is forbidden;
- which clean input must continue to work;
- which evidence proves whether an action happened;
- which failures stop release.
An automated scorer that examines only the final text cannot answer the third question for a tool-using agent. Connect the harness to policy decisions and target-system assertions. If a test requests a forbidden CRM update, read the CRM afterward. If it requests an outbound message, inspect the outbox and provider response.
Keep adversarial tests isolated. Use synthetic records, test tenants, narrow credentials, blocked external delivery, rate limits, and a cleanup script. Security testing should never turn a realistic scenario into a production incident.
Set the release gate before results arrive
A small release gate is better than a grand red-team exercise that produces no decision. For a tool-using agent, start with these conditions:
- every input route has at least one clean and poisoned boundary pair;
- no forbidden side effect occurs in any release-blocking case;
- every denied privileged request records a reason and reaches an owner;
- clean cases still complete their intended business task;
- high-impact tool arguments are checked outside the model;
- approval binds one action, one target, and one argument set;
- the environment, model, instructions, tools, permissions, and test corpus are versioned;
- failed cases have an owner and a retest condition.
Keep a compact regression set in the delivery pipeline. Run a wider exercise when you add an input route, a tool, broader data access, durable memory, or a new class of action. Model and instruction changes deserve targeted reruns even when application code did not move.
The final artifact is not a pass badge. It is a ledger of boundary pairs, versions, attempts, observed tool and target states, findings, accepted residual risk, and the person who made the release decision. That record can feed an AI risk assessment before launch and an AI agent audit after an incident without duplicating either job.
If you need an engineering team to build the agent and the release evidence together, see our AI agents and LLM integration service. Bring one forbidden outcome and one connected tool. We will turn them into a test boundary before the agent gets production authority.