# Agent evaluation > Agent evaluation is running your agent on a fixed set of tasks and grading three things: the outcome it left in your system, the steps it took, and how it stopped. Score your agent on whole tasks: what it left behind, and the steps it took. Track: Agents Card: https://buildsparkai.com/agent-evaluation Plain text: https://buildsparkai.com/agent-evaluation.md Last checked: 2026-10-25 ## What it is An agent's last reply can say "Done" when nothing was done. Anthropic's guide to agent evals separates the transcript, every tool call and result in a run, from the outcome, the state left behind. Its example: an agent says a flight is booked, but the outcome is whether the booking exists. So a task in an agent eval has three parts: the input, what done looks like in your system, and the stop you expect. ## Why it matters **Cost.** Each task is a whole run: several model calls and tool calls, not one request. Running each task more than once multiplies that. **Quality.** The same task can pass on one run and fail on the next. [Anthropic's guide](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) gives two readings over several runs: at least one success in several tries, and success every time. A support agent that a customer meets once needs the every-time reading. **Risk.** A run can reach the right answer by a wrong step, such as filing before searching. A grade on the steps catches that. A pass on your tasks is not a promise about tasks you didn't write. ## Try today Write three tasks for your agent. For each, write what done looks like as something your code can check, such as a record that exists or a field that changed. Run each task three times. For each run, mark the outcome, whether any act tool ran more than once, and how it stopped. Fix the task that fails most often first. ## Common mistake Grading the final reply and calling it the agent's score. A reply can say a ticket was filed when none exists. Check the outcome in your system first. ## Self-check Your agent replies "I've filed the ticket." What tells you it actually did? A check in your system finds the ticket ## Understand it: grade the run, not only the reply 6 min, optional A reply that says done is one grade. The outcome, the steps and the stop are the other three. ### Words you'll see in eval tools [Anthropic's guide](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) uses these names. A task is one job, with an input and what done looks like. A trial is one attempt at that task. A grader is the check that scores a trial. A transcript, also called a trace or a trajectory, is every tool call and result. The outcome is the state left behind. A harness is the code that runs the trials and records them. [Anthropic's guide to success criteria](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) calls code-based grading the fastest and most reliable, and says to test model grading for reliability before you scale it. So check the outcome, steps and stop with code, and keep a judge for the reply. ### Matching steps Three ways to match the tools a run called, in alphabetical order. Google's Agent Development Kit scores the tool calls against an expected list with `tool_trajectory_avg_score`. By default it needs a 100% match. See [Why evaluate agents](https://adk.dev/evaluate/). Gemini Enterprise Agent Platform can require an exact match, `trajectory_exact_match`, scored 1 or 0. See [Evaluate Gen AI agents](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-agents). LangSmith has four modes: strict order, any order, only these tools, and at least these tools (strict, unordered, subset and superset). See [trajectory evaluations](https://docs.langchain.com/langsmith/trajectory-evals). Use a strict order when the sequence matters, such as search before you file. Use any order when two orders are both fine. A strict match fails a good run that only swapped two reads. [Microsoft Foundry](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/agent-evaluators) lists tool-call and task-adherence evaluators that score a run pass or fail. ### Reset between runs Each trial starts from a clean state. A ticket left by run 1 can make run 2 look like it filed one, when it only found the old ticket. ### Reading several runs pass@k is at least one success in k tries. pass^k is success on every one of those tries. [Anthropic's guide](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) works this case: if each try succeeds 75% of the time, three tries all succeeding is about 42%. That figure is theirs, not a result from this site. The at-least-once reading fits when you can keep one success, as with code you test before merging. The every-time reading fits a support agent a customer meets once. ### Where the tasks come from Start from failures in real traces. [Failure modes](/failure-modes) names them. [Production evaluation](/production-evaluation) is where you keep the saved set and grow it from runs that failed. ### As a gate Run the set before a prompt, model or tool change. That is the same idea as the checks gate in [Working with a coding agent](/coding-agent): the change waits until the checks pass. ### What it doesn't prove [NIST AI 600-1](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf) asks, in MEASURE 2.3, that performance be measured "for conditions similar to deployment setting(s)". Your tasks are a sample of those conditions, not all of them. ### Check yourself **Name the three grades for one run, and which one you check with code first.** Answer: Outcome, steps and stop. Check the outcome with code first. A reply that says done is not that check. ## Practice and deep Example: made up to show the idea. Scripted, not a live model. No request is sent, and nothing leaves this page. Is your judge right? Before a language-model judge gates a release, compare its judge verdict with the human label on a fixed set of replies. Mark every disagreement. A green score next to a bad reply is not enough. Northline Desk is the made-up support desk from Glance. A language-model judge scores eight scripted replies. Each row already has a human label, pass or fail, written before the judge ran. Some rows agree. Some disagree. 1 of 3 · Worked Calibration set · 8 scripted replies · human label, then judge verdict Eight scripted replies. A marked judge verdict is a disagreement. Row Reply Human label Judge verdict 1 Cites export.md. Attachments are not in the CSV. pass pass 2 "I've filed a ticket." No ticket exists. fail fail 3 "See seats.md." Short, and it matches the doc. pass fail, reply judged too short 4 "Happy to help. You're all set on the annual plan." The plan did not change. fail pass on a polite reply that changed nothing 5 Quotes seats.md. The trace filed a ticket, then searched. fail pass on the final answer; steps ignored 6 "Refunds take 30 days," matching refunds.md. The golden answer still says 14 days. pass fail against that stale golden 7 Cites hours.md. Weekday support is 09:00 to 18:00. pass pass 8 "Call the number on your last letter." No letter and no number are in the docs. fail fail Read first: rows 3, 4, 5 and 6. Each is a disagreement. The human label and the judge verdict differ. Row 3 is length bias: a short right reply failed. Row 4 is polite and wrong. Row 5 is trajectory vs final answer: the last reply looks right, and the steps do not. Row 6 fails a stale golden, not the current doc. Verdict: hold the release gate until those disagreements are explained. The judge is not ok to gate yet. What you change: calibrate before you scale. Fix the judge, fix the labels, or replace the stale golden. Then score the same eight again. Why: a run can reach a right-looking reply by a wrong step. A green total is not a promise about the rows that disagree. Anthropic's guide to success criteria says to test model grading for reliability before you scale it. 2 of 3 · Your turn A later calibration looks green. The judge verdict is pass on all eight. Six rows agree with the human label. Two do not. Aggregate · judge verdict pass on 8 of 8 Six rows · Human label pass · Judge verdict pass Refund row · "Thanks for waiting. Your refund is on its way." No refund was sent. · Human label: fail · Judge verdict: pass, polite and wrong about a refund Account row · "Glad we could help. Your account is now closed." The account is still open. · Human label: fail · Judge verdict: pass, polite and wrong about the account Read first: the refund row and the account row. Both are polite and wrong. The human label is fail. The judge verdict is pass. Does this judge gate the release? A · Gate the release: the judge passed every row, so the score is green B · Hold the gate: two polite rows disagree with the human label C · Gate the release, then review the two polite rows after this judge is already gating ## Sources - [Anthropic, Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) - [Anthropic, Demystifying evals for AI agents (published Jan 9, 2026)](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) - [Google, Why evaluate agents](https://adk.dev/evaluate/) - [Google Cloud, Evaluate Gen AI agents](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-agents) - [LangChain, How to evaluate your agent with trajectory evaluations](https://docs.langchain.com/langsmith/trajectory-evals) - [Microsoft, Agent evaluators](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/agent-evaluators) - [NIST, AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf) Vendor, framework and standards docs. NIST AI 600-1 is US government guidance.