# Production evaluation

> Production evaluation is a saved set of examples and graders you run again before a prompt, model, or retrieval change faces users.

A repeatable eval set you run on changes that will face real users.

Track: Ship & Maintain
Access: free
Card: https://buildsparkai.com/production-evaluation
Plain text: https://buildsparkai.com/production-evaluation.md
Last checked: 2026-10-18

## What it is

A production eval is the set you keep, not one look at a single reply. LangSmith's evaluation docs describe offline evaluation as running a version over a curated dataset before deployment, so you can compare versions before users see the change. A dataset is a collection of examples. Each example is an input and an optional reference output. An experiment is one application version run on that dataset, so you can compare prompts or models on the same examples. Anthropic's guide says to design evals that mirror the real task, including edge cases.

## Why it matters

**Cost.** Running the set calls the model on every example. If the grader is an LLM-as-judge, LangSmith describes that as another model call. This card does not add a price.

**Quality.** A change that helps one ticket can fail another that used to pass. LangSmith describes regression tests as checks that a new version does not degrade cases the current version handles. Anthropic says the set should mirror the real task, including edge cases.

**Risk.** A set that never grows misses the cases users actually send. LangSmith describes a feedback loop where failed production runs become candidates for dataset examples. A pass on the set is not a promise about tickets that are not in it.

## Try today

Pick ten real requests you already know how to grade. Save the input and the mark. Run them once and write down the passes. The next time you change the prompt, the model, or the retrieved documents, run the same ten before users see the change.

## Common mistake

Reading one new reply, deciding it looks better, and shipping. The saved set is what shows which old cases got worse.

## Self-check

What should you do with the same eval set when you change the prompt?

Run the saved set again before the change faces users

## Practice and deep

Coming soon. Ten saved tickets, run before and after a prompt change, with the new failures kept.

Coming soon. How LangSmith separates a dataset of examples from an experiment that runs one version on that dataset, and how Anthropic says to include edge cases.

## Sources

- [LangSmith, Evaluation concepts](https://docs.langchain.com/langsmith/evaluation-concepts)
- [LangSmith, Evaluation types](https://docs.langchain.com/langsmith/evaluation-types)
- [Anthropic, Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)

Opened 2026-10-18. Vendor, framework, or standards docs. This card does not add a rate, a usage count, or a score of its own. Numbers quoted from a source are labelled vendor-reported or as that document's own words.