BuildSpark AI

Map · Ship & Maintain · 01 · Production evaluation

Concept card · Ship & Maintain

Production evaluation

Production evaluation is a saved set of examples and graders you run again before a prompt, model, or retrieval change faces users.

A repeatable eval set you run on changes that will face real users.

Last checked 2026-10-18 Free card 5 min

Ask about Production evaluation

What changed

Oct 18: First card. The picture reruns one small set after a prompt change. Sources are LangSmith's evaluation docs and Anthropic's eval guide.

1 What it is

A production eval is the set you keep, not one look at a single reply. LangSmith's evaluation docs describe offline evaluation as running a version over a curated dataset before deployment, so you can compare versions before users see the change. A dataset is a collection of examples. Each example is an input and an optional reference output. An experiment is one application version run on that dataset, so you can compare prompts or models on the same examples. Anthropic's guide says to design evals that mirror the real task, including edge cases.

Illustration. Three made-up tickets, graded pass or fail. The words are written for this picture. They are not a live model and not a score from a product.

Saved set: close time, return window, and Sunday parking.

Close timePass. The reply says 18:00.

Return windowPass. The reply says 30 days.

Sunday parkingPass. The reply says the source does not say.

The picture's set passes. That is this drawing's starting point, not a measured score.

2 Why it matters

Cost

Running the set calls the model on every example. If the grader is an LLM-as-judge, LangSmith describes that as another model call. This card does not add a price.

Quality

A change that helps one ticket can fail another that used to pass. LangSmith describes regression tests as checks that a new version does not degrade cases the current version handles. Anthropic says the set should mirror the real task, including edge cases.

Risk

A set that never grows misses the cases users actually send. LangSmith describes a feedback loop where failed production runs become candidates for dataset examples. A pass on the set is not a promise about tickets that are not in it.

3 How to apply · try today

Pick ten real requests you already know how to grade. Save the input and the mark. Run them once and write down the passes. The next time you change the prompt, the model, or the retrieved documents, run the same ten before users see the change.

Common mistake

Reading one new reply, deciding it looks better, and shipping. The saved set is what shows which old cases got worse.

Self-check

What should you do with the same eval set when you change the prompt?

Questions

What is production evaluation?
Production evaluation is a saved set of examples and graders you run again before a prompt, model, or retrieval change faces users.
Is a production eval the same as reading one reply?
A production eval is a saved set you can run again. One fluent reply is a sample. Anthropic's guide says to design evals that mirror the real task, including edge cases.
Do I need a vendor eval product to keep this set?
No. LangSmith describes offline evaluation as a dataset of examples and an experiment that runs one version on that dataset before users see the change. You can keep the same inputs and marks in a file you run yourself.