How to evaluate AI agents

The components of an evaluation

Anthropic defines an AI evaluation as a test that measures a system by giving it an input and scoring its output. For agents, this test can be broader than examining a single response. An agent can use tools over multiple turns, change the state of the environment, and adapt its behavior based on intermediate results.

Each task in an evaluation is a test with defined inputs and success criteria. Each attempt at the same task is called a trial. Because model outputs can vary from one run to another, Anthropic runs multiple trials to obtain more consistent results.

A grader is logic that scores one aspect of agent performance. A task can contain multiple graders, and each grader can contain multiple assertions. Scoring can use a weighted structure that requires grader scores to reach a threshold. A binary structure that requires all grader results to pass, or a combination of the two, can also be used.

A transcript is the complete record of outputs, tool calls, intermediate results, and other interactions that occur during a trial. The outcome is the state of the environment at the end of the trial. In Anthropic's flight booking example, the transcript may say that a reservation was made; the outcome is whether a reservation actually exists in the environment's SQL database.

An evaluation harness is the infrastructure that runs the evaluation from start to finish. It provides instructions and tools, runs tasks concurrently, records steps, scores outputs, and brings the results together. An agent harness is the system that enables the model to act as an agent: it processes inputs, orchestrates tool calls, and returns results. According to Anthropic, when an agent is evaluated, the model and the agent harness are evaluated together.

Defining tasks and success criteria

Anthropic recommends that evaluation tasks have clear success criteria. One measure of task quality is whether two domain experts can independently reach the same pass or fail decision for the same task. Ambiguity in the task description appears as noise in the metrics. An ambiguous rubric prepared for a model-based grader can also lead to inconsistent judgments.

An agent that follows the instructions correctly should be able to pass the task. Anthropic gives an example in which an agent is asked to write a script but is not given a file path. If the tests assume a particular file path, the agent may fail through no fault of its own. The conditions checked by the grader need to be clear in the task description.

Anthropic also finds it useful to create a known-working reference solution for each task that passes all graders. This solution shows that the task is solvable and verifies that the graders are configured correctly. A reference solution is not a transcript definition that requires the agent to reach the same result by exactly the same path.

An evaluation suite is a set of tasks organized to measure specific capabilities or behaviors. In Anthropic's customer support example, a suite may contain tasks related to refunds, cancellations, and escalation to another authority. Each task has its own input and success criterion; the suite groups these tasks under a common purpose.

Anthropic also distinguishes between a capability eval and a regression eval. A capability eval measures what work the agent can do and targets tasks the agent struggles with. A regression eval measures whether the agent can still perform tasks it has previously completed. Once an agent starts regularly passing tasks in a capability eval, those tasks can be used in a regression suite.

Types of graders

Anthropic describes three types of graders for agent evaluations: code-based, model-based, and human. Each can evaluate part of the transcript or outcome. Anthropic recommends using a deterministic grader where possible, an LLM grader when needed or for additional flexibility, and human graders for additional verification.

Code-based grader methods include string match, binary test, static analysis, outcome verification, and tool call verification. Tool call verification can inspect the tools used and their parameters. Transcript analysis can address values such as the number of turns or token usage.

Anthropic describes code-based graders as fast, objective, and repeatable. They can verify specific conditions and are easy to debug. However, they can be brittle when faced with valid changes that do not exactly match the expected pattern. They are also limited in their ability to evaluate some nuanced or more subjective tasks.

Model-based grader methods include scoring with a rubric, natural language assertions, pairwise comparison, reference-based evaluation, and consensus among multiple evaluators. Anthropic states that these graders can evaluate open-ended tasks and free-form outputs. Their results are not deterministic; they require calibration with human graders for accuracy.

Anthropic recommends preparing a clear and structured rubric for each dimension in LLM-as-judge evaluation. Rather than having a single evaluator score all dimensions, each dimension can be evaluated by a separate LLM-as-judge. Giving the evaluator the option to return “Unknown” when there is insufficient information is another method Anthropic describes.

Human grader methods include expert review, sampling-based review, and inter-rater agreement. Anthropic uses human evaluation to calibrate model-based graders. This approach can take time and may require access to experts at scale.

Evaluation based on the agent's work

For an agent that writes code, Anthropic focuses on clearly defining tasks, keeping the test environment stable, and thoroughly testing the generated code. Unit tests can be used for correctness, and an LLM rubric for overall code quality. Behaviors such as tool calls or interaction with the user can also be evaluated by model-based graders using clear rubrics.

SWE-bench researchers give the model a codebase and a problem description; the model then produces a .patch. During evaluation, the test patch and the model-generated patch are applied to the codebase, and then the test script is run. For the task to count as solved, all tests in the FAIL TO PASS and PASS TO PASS groups must pass. A missing or failing test is considered a failure.

Before including tasks in the evaluation, SWE-bench verifies that the solution patch and test patch can be applied to the codebase and that the tests can be run. The researchers compare test results before and after the solution is applied. A task is removed from the evaluation set if it does not meet the condition that at least one test fails before the solution and passes afterward.

For an agent that conducts conversations, Anthropic uses verifiable end states together with rubrics that address both task completion and interaction quality. In a support example, whether the ticket is resolved and the refund has been processed can be evaluated through the state. Communication characteristics, such as the clarity of the explanation, can be included in the rubric.

The output of an agent that conducts research may be an answer or a report. According to Anthropic, criteria such as comprehensiveness and accuracy in this kind of work depend on the context of the task. A groundedness check examines whether claims are supported by the referenced content. A coverage check defines the key facts that a good answer should include.

Computer use agents interact with software through screenshots, mouse clicks, keyboard inputs, and scrolling. Anthropic describes running these agents in a real or sandbox environment and examining whether the intended outcome was achieved when evaluating them. For a task that modifies data, the backend state can also be verified, so the evaluation does not rely only on whether a confirmation appears on screen.

Repetitions and interpreting results

Because agent behavior can vary between runs, Anthropic runs multiple trials for each task. Each trial must start from a clean environment. Files left over from previous runs, cached data, or other shared state can lead to errors unrelated to agent performance or artificially high performance.

Anthropic defines two different repeat metrics. pass@k measures the probability that at least one of k attempts produces a correct solution. pass^k measures the probability that all k trials succeed. In the first metric, a single success is enough; in the second, all trials must succeed. Anthropic states that usage depends on product requirements.

Transcript review helps explain the cause of a failure. According to Anthropic, the transcript can show whether the agent made a mistake or the grader rejected a valid solution. Ambiguous task descriptions, grader errors, and constraints on the agent harness can also produce low scores despite good agent performance.

Limitations

According to Anthropic, a capability eval in which all solvable tasks are passed can track regressions, but it leaves no room to measure improvement. Automated evaluation is also not the only view of agent performance. Anthropic includes monitoring the production environment, user feedback, transcript review, and systematic human evaluation among the components of understanding performance.