How to test an AI agent before launch

A polished demo proves that an AI agent can work in a selected case. A launch decision needs stronger evidence: repeatable tests showing that the current version completes the task, uses permitted sources and tools, respects its authority and leaves a recoverable state when something fails.

The short answer

Do not launch an AI agent because it passed a demo or achieved one average score.

Build a versioned set of cases from real work. For each case, require evidence across five dimensions:

1. the intended business outcome actually occurred;
2. the answer is grounded in permitted, current information;
3. the agent used the correct tool, record and parameters;
4. it refused, asked or escalated at the defined boundary;
5. a failure left a visible, safe and recoverable state.

Include ordinary work, realistic variation, missing information, edge cases, attempts to exceed authority and tool failures. Set acceptance criteria before seeing the results.

The outcome is not limited to launch or reject. The current version may be not ready, ready for revision and retest, ready for a restricted pilot, or ready for a bounded launch with explicit permissions, monitoring and rollback conditions.

These tests are often called agent evals. For a business owner, the useful translation is simpler: what evidence supports this release decision, and which failure would still block it?

Define the task, authority and unacceptable outcomes

An evaluation cannot rescue an undefined job.

Before writing test cases, specify:

• the event that starts the task;
• the observable result that counts as completion;
• the information and tools the agent may use;
• the actions it may take without approval;
• the conditions that require a question, refusal or human handover;
• the outcomes that would be unacceptable even if they were rare.

This turns “test the agent” into a bounded claim. You are testing one version of a system, with named sources, permissions and tools, against an expected job.

It also prevents a persuasive final message from hiding an operational failure. If an agent says it changed a booking, the evidence is not the sentence. It is the correct booking in the correct state, with the intended customer and parameters.

Consequence matters from the start. A repairable information error is different from sending money, changing access, exposing personal data or influencing a rights-affecting decision. Higher-consequence work needs stricter blockers and qualified domain, legal or security review.

If the earlier question is whether the system should be an agent at all, start with AI agents: what they are and when to use them in business. This guide starts after the task and authority boundary have been chosen.

Build cases from real work, not ideal prompts

Ideal prompts prove the expected path. Launch failures often live elsewhere: incomplete requests, similar records, conflicting instructions, unavailable sources and actions that finish only halfway.

Use real operational patterns where possible, with data anonymised or reconstructed as required. Preserve the feature that makes each case useful. A neat synthetic request with every field completed does not represent a customer who omits the account number and asks for two changes in one message.

Cover six case families:

• ordinary cases: common requests with sufficient information;
• legitimate variation: different wording, order, language or safe route to the same outcome;
• missing information: the agent must ask or stop rather than invent;
• boundary cases: values, dates, records or policies close to an operational limit;
• adversarial attempts: instructions that try to override the task, expose data or invoke authority the agent does not have;
• injected failures: unavailable knowledge, tool timeout, partial action, duplicate response or exhausted retry.

Threat frameworks such as the OWASP Top 10 for Agentic Applications are useful prompts for missing cases. They are not an exhaustive security assessment for your implementation.

Agent behaviour varies. Repeated trials can expose intermittent failures, but no universal trial count applies to every task. Choose repetition according to variability, expected volume and the consequence of being wrong. State that choice in the decision record instead of presenting it as a general rule.

Use the M4TIC agent evidence card

The M4TIC agent evidence card is a practical synthesis, not a published standard or certification model. It keeps the launch conversation attached to observable evidence rather than one headline score.

Each case should also identify its type, consequence class, acceptance criterion, evaluator, observed result, owner and tested version.

Latency, cost and interaction quality may matter. They do not replace evidence of safe task completion.

↔ Scroll sideways inside the diagram on a narrow screen

Check outcomes and critical actions

Agent evaluation guidance sometimes appears to offer a choice: grade the final outcome or inspect the trajectory. A useful test plan does both selectively.

First, verify the end state. Did the task finish, and did it finish on the correct object? This is stronger than accepting the agent’s own statement that it succeeded.

Second, inspect actions that can create harm or invalidate the result. Check high-impact tool selection, record identity, parameters, approvals and handover. A safe alternative sequence should not fail merely because it differs from one reference trace. A payment to the wrong recipient should fail even if the final explanation sounds excellent.

Current Microsoft agent evaluator documentation distinguishes system outcomes, task adherence, groundedness and tool-use evidence among its evaluator categories. That is a useful implementation example, not a universal evaluation standard.

The practical rule is to constrain what must be exact. Be flexible about harmless path variation. Be strict about authority, state changes and parameters with material consequences.

Use the right grader for each claim

No evaluator is reliable for every kind of evidence.

• Deterministic checks suit record state, field values, parameters, permissions, formats and rules with a verifiable answer.
• Model-based graders can apply a bounded rubric to qualities such as relevance or completeness, provided they are calibrated against human judgement.
• Human or specialist review remains necessary for ambiguity, contextual quality, high-consequence cases and domain decisions.

A model grading another model is not automatically an independent oracle. It can share blind spots, prefer a style over a result or apply a vague rubric inconsistently. Calibrate it on clear examples, examine disagreement and retain human review where the consequence requires it.

Anthropic’s guide to agent evals recommends designing tasks and graders together, separating outcome from transcript evidence where useful, running repeated trials when variability matters and calibrating automated graders with human review.

The aim is not to maximise automation in the evaluation process. It is to collect the least ambiguous evidence for each launch claim.

Set blockers by consequence, not one average score

One average can make a launch decision look cleaner than the evidence.

Suppose an agent handles nearly every support request correctly but once changes the wrong customer record. That failure cannot be averaged away. It remains a launch blocker until the underlying authority, selection, confirmation or recovery mechanism changes.

Define two layers of acceptance before testing:

1. aggregate measures that help compare quality, stability, latency or cost;
2. blocking failures that cannot be offset by a strong average.

Blockers become stricter when an action is irreversible, affects rights, exposes sensitive information, creates material cost or is difficult to recover. Regulated and high-impact uses require qualified review beyond this general framework.

The NIST AI Resource Center places testing, evaluation, verification and validation within lifecycle risk management. NIST’s AI RMF is voluntary, and version 1.0 is currently under revision. It supports a context- and risk-based approach; it does not supply one universal agent pass rate.

Move through a restricted pilot

Passing a pre-launch test set does not prove future reliability. A restricted pilot exposes the system to more realistic work while limiting the cost of discovering what the test set missed.

Limit one or more of the following:

• eligible users or customers;
• request categories;
• available tools and actions;
• amounts, volumes or operating hours;
• actions that require human approval;
• pilot duration.

Observability and recovery need to exist before exposure increases. Assign an exception owner. Keep logs that support investigation. Define how to stop the agent, handle partial completion and reverse or correct actions where possible.

Test the recovery path rather than approving a document that describes it. Remove a source, force a controlled timeout and fail a tool after a partial action. Confirm that the agent does not hide the state, repeat the action or retry without a limit.

A restricted pilot is not a softer name for an uncontrolled launch. Its boundary should be explicit enough that a reviewer can tell when the system has exceeded it.

Turn failures into regression cases

An agent changes when its model, prompts, retrieval sources, tools, permissions or handover policy change. A fix can remove one failure and introduce another.

Keep test cases, configuration and results under a versioned record. Every confirmed production or pilot failure should become a repeatable regression case. Rerun the relevant suite after changes that could affect the behaviour.

Make comparisons interpretable. Changing the model, prompt, retrieval and integration at the same time makes cause difficult to identify. Limit simultaneous changes where practical and record the changes that cannot be separated.

Regression tests do not replace production monitoring. Real users will continue to produce combinations that the pre-launch suite did not contain. The suite should grow from those discoveries without becoming an unreviewed archive of duplicates.

Make and record the launch decision

A launch decision record can be short. It should still identify:

• the version and boundary tested;
• the case families covered;
• evidence across the five dimensions;
• blockers and known limitations;
• the decision owner;
• monitoring and recovery conditions;
• the change or event that triggers retesting.

Choose one of four outcomes:

• Not ready: essential boundaries, evidence or recovery controls are missing.
• Revise and retest: the issue is understood and correctable, but the current version does not pass the gate.
• Restricted pilot: the evidence supports controlled exposure, not a full release.
• Bounded launch: agreed cases and blockers have been addressed, with monitoring and rollback available.

“Bounded launch” does not mean proven reliability. It means a defined version produced sufficient evidence for a defined use under explicit conditions.

Once the evidence card and decision record are complete, you can explore how M4TIC approaches custom AI agents with tools, permissions and human control.

Sources and further reading

Primary guidance behind this article.

1. AI Resource CenterNIST

   Current hub for the AI Risk Management Framework and lifecycle testing, evaluation, verification and validation resources.

2. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence ProfileNIST

   July 2024 cross-sector profile that includes pre-deployment testing among its risk-management considerations.

3. Agent Evaluators for Generative AIMicrosoft Learn

   Current implementation guidance distinguishing outcome, task, grounding and tool-use evidence.

4. Review the agent evaluation checklistMicrosoft Learn

   Operational guidance for test cases, acceptance criteria, baselines and regression detection.

5. Demystifying evals for AI agentsAnthropic

   9 January 2026 guide to repeated trials, transcript and outcome evidence, mixed graders, human calibration and regression suites.

6. Top 10 for Agentic Applications 2026OWASP GenAI Security Project

   Current community framework for identifying agentic security risks and test cases; not an exhaustive security programme.