Evaluate AI Agents

The Evaluate tab is a space for systematic AI agent testing. You can create test cases, run bulk evaluations, and review results in a drill-down view.

The Build tab's Test panel is best for quick ad hoc testing. The Evaluate tab lets you run your full test suite at once and compare results across versions.

For ad hoc testing, see Create and Configure an AI Agent. For monitoring, see Monitor AI Agents. For best practices, see Best Practices for Designing AI Agents.

What is an evaluation?

An evaluation is a bulk execution session. It runs multiple test cases against a specific AI agent version and records the results for review.

Key terms used in the Evaluate tab:

• Test case: A saved input-and-expected-output pair that defines a scenario to test. The expected output is reference text, not an exact-match rule.
• Evaluation: A bulk run of one or more test cases against a selected agent version.
• Reference text: The expected output field in a test case. Use it to spot where the agent's response differs.

Manage test cases

Test cases define the scenarios you use to test your AI agent. Each test case has input values and reference text to compare against the agent's response.

Prerequisites for managing test cases

You must have Editor or Admin permissions on the AI agent to create, edit, or delete test cases.

Create a test case

To create a new test case:

1. In the Evaluate tab, click Manage Test Cases.
2. In the Manage Test Cases modal, click New Test Case.
3. Enter a name for the test case that describes the scenario.
4. Fill in values for each input variable in the agent's config.
5. Enter reference text in the expected output field. Use this text as a visual baseline when reviewing results.
6. Click Save.

Edit a test case

To update an existing test case:

1. In the Evaluate tab, click Manage Test Cases.
2. Select the test case you want to modify.
3. Update the input values or reference text as needed.
4. Click Save.

Delete a test case

To remove an obsolete test case:

1. In the Evaluate tab, click Manage Test Cases.
2. Select the test case you want to remove.
3. Click Delete.
4. Confirm the deletion.

Test case behavior when AI agent inputs change

Test cases persist at the AI agent version level. When you change the AI agent's input or output fields, existing test cases update as follows:

• Deleted input or output: The field is removed from existing test cases.
• Added input or output: The new field appears blank. Edit each test case to add values for new fields.
• Changed field type: The system tries to convert values. This can fail silently. Review test cases after type changes.

Run an evaluation

An evaluation runs your test cases against a specific AI agent version and records the results.

Prerequisites for running an evaluation

You must have Editor or Admin permissions on the AI agent to run evaluations.

Configure and start an evaluation

To run an evaluation from the Evaluate tab:

1. In the Evaluate tab, select the AI agent version from the version selector. It defaults to the latest version.
2. Select which test cases to include. The default is all test cases. The system remembers your last selection.
3. Set runs per case (1–3). The default is 1. The system remembers your last setting.
4. (Optional) If you want to note the purpose of this run, enter a description.
5. Click Run Evaluation.

The evaluation runs test cases against the chosen version. Each test case runs on its own.

Stop an evaluation

To cancel all runs in an active evaluation, click Stop in the progress bar. Stopping cancels remaining runs. Finished results stay available.

Review evaluation results

The Evaluate tab has a three-level drill-down for results. Each level adds more detail.

Level 1: Evaluation grid

The evaluation grid shows all past runs for the agent. Each row shows:

Column	Description
Evaluation ID	Unique ID for the run. Click to drill into results.
Version	The agent version that was evaluated.
Date	When the evaluation was started.
Test count	Total test case runs (test cases times runs per case).
Average duration	Average run time across all test cases in the evaluation.
Average AI actions	Average AI actions consumed per run.

Level 2: Evaluation detail

Click an evaluation ID to view the test cases in that run. The detail view shows each test case with its status and metrics.

Level 3: Test case comparison

Click a test case in the detail view to open the comparison layout. It uses three columns:

Column	Content
Input	The input values provided to the agent (read-only).
Expected output	The reference text you defined in the test case.
Actual output	The agent's response for the run.

Compare the two output columns to find where the agent differs from what you expect.

Provide developer feedback

Developer feedback captures your quality judgments on test results. Feedback gives you data to track agent quality over time. It replaces informal "vibe checks" with stored, structured ratings.

Feedback workflow

After reviewing a test result in the comparison view, give feedback in up to three steps:

1. Click thumbs up or thumbs down (required). Positive feedback (thumbs up) marks the run as a strong example.
2. (Optional) If you chose thumbs down, select one or more checkboxes to identify what went wrong.
3. (Optional) If you want to add notes, enter a free-text comment.

Feedback targets

Developer feedback targets the overall outcome of the AI agent run. Give feedback in the evaluation detail view or test case comparison view.

Failure categories

Selecting one or more checkboxes after a thumbs-down rating helps you identify what went wrong and surface recurring failure patterns. Select all that apply:

Category	Use when
Wrong output	The agent's final answer is incorrect or incomplete.
Made up information	The agent made up facts not in its tools or inputs.
Too many steps	The agent took a longer path than needed.
Got stuck	The agent looped or failed to make progress.
Took too long	The agent took too long to produce a result.
Failed to recover from error	The agent encountered an error and could not continue.
Other	The issue does not fit the other categories.

Strong examples

Strong examples are agent runs that show ideal outcomes. Mark a run as a strong example when it has the quality, tone, and depth you want in production.

To create a strong example:

1. In the test case comparison view, review the actual output.
2. Click thumbs up.

Positive feedback (thumbs up) marks the run as a strong example. Mark several runs as strong examples to show the range of good agent behavior. A library of strong examples helps you spot regressions in future evaluations.

Feedback from multiple developers

When multiple developers rate the same run, all ratings are stored. The system averages all feedback when it calculates accuracy. No single rating overrides another.

Where to provide feedback

You can provide developer feedback in three locations within Agent Studio:

• Evaluation detail view: Agent-level feedback only. Rate the overall outcome.
• Ad hoc test pane (Build tab): Rate results from test runs during development.
• Monitor grid drill-down: Rate results from production or past runs.

Feedback from all locations feeds into the same accuracy metrics.

Evaluation metrics

Each evaluation shows key metrics that signal agent quality at a glance.

• Total tests: Test cases times runs per case. For example, 10 test cases with 2 runs each = 20 total tests.
• Overall accuracy: Thumbs-up count divided by total ratings. No score shows if no feedback exists yet.
• Average test time: Average run length across all test cases.
• Average AI actions per test: Average AI actions used per run.

Per-test-case accuracy

Each test case card shows its own accuracy score. This is the average across all runs and all feedback for that test case.

Track metrics over time. A drop in accuracy or a rise in test time after a change means something needs a closer look.

Evaluation security

Access to the Evaluate tab depends on your permission level for the AI agent.

Permission level	Evaluate tab access
Viewer	Can access the Evaluate tab and view test cases and the monitor grid. Cannot drill down into evaluation details or initiate an evaluation run.
Editor	Can view evaluations, create and run evaluations, and provide feedback.
Administrator	Can view evaluations, create and run evaluations, and provide feedback.
Initiator	Cannot access evaluations.

Evaluation limitations

Keep the following limits in mind when using the Evaluate tab:

• Runs per case: Maximum 3 runs per test case per evaluation.
• Test case persistence: Test cases persist at agent version level. If inputs change, update test cases to match.
• Test isolation: Agents with side effects (creating records, sending emails) can give different results across runs. Use repeat-safe test inputs when possible.
• Manual curation only: Automated test case generation is not available.
• No automated scoring: LLM-as-judge automated evaluations are not available. Use developer feedback (thumbs up/down) to build accuracy scores.
• No cost estimation: Cost estimation before execution is not available.

Related resources

• Create and Configure an AI Agent for ad hoc testing in the Build tab.
• AI Agent Reference for security details and run limitations.
• Monitor AI Agents for production run monitoring.
• Best Practices for Designing AI Agents for test case design guidance.

Feedback