Cost
Each example is a model request, and a judge is another request on the reply. Anthropic's guide says human grading is slow and expensive, and that an LLM grade is the scalable option after you have checked it. This card does not add a price.
Map · Foundations · 08 · Evaluation basics
Concept card · Foundations
Evaluation basics
Evaluation basics is checking a model on examples you already know how to grade, including with another model as the judge, before you trust a change.
Checking a model on examples you already know the answer to, before you trust a change.
What changed
Oct 18: First card. The picture grades one reply you already know. LLM-as-a-judge is a section of this card. Sources are Anthropic's eval guide and LangSmith's LLM-as-a-judge guide.
1 What it is
An eval is a set of examples you can grade. Anthropic's guide to success criteria and evaluations says a successful LLM application starts by defining what success is, then designing evaluations that measure performance against it. The same guide says to automate grading when you can, with code or with an LLM, and that more graded questions beat a handful of replies you only read by hand. Code-based grading is the fastest and most reliable when a rule can decide. Human grading is the most flexible, and the guide calls it slow and expensive. LLM-based grading is the option for a judgment a rule cannot make, after you have tested that the judge is reliable. That second model is what this card calls LLM-as-a-judge.
LLM-as-a-judge
LLM-as-a-judge means a second model grades the reply. Anthropic's guide says to give that grader a clear rubric, a specific mark such as correct or incorrect or a scale from 1 to 5, and room to reason before the score. Test it on examples a person has already graded before you scale it. LangSmith's LLM-as-a-judge guide says many LLM replies have no single correct answer, so a judge scores them against a rubric you write. That rubric can be a true or false mark, a category, or a number. The same guide says you can collect a person's correction of the judge and feed those corrections back as examples.
2 Why it matters
Quality
A change that looks better on one reply can be worse on the set. Anthropic says to design evals that mirror the real task, including edge cases, so you can see whether a prompt change helped before users see it.
Risk
A judge can pass a reply a person would fail, or the reverse. Anthropic says to test LLM grading for reliability before you scale it. LangSmith's guide describes collecting human corrections so the judge stays closer to those marks. A passing eval is not proof the reply is true outside the examples.
3 How to apply · try today
Write five questions you already know the answer to. Run your prompt. Mark each reply pass or fail yourself. If you use a model as the judge, compare its marks with yours on those five before you trust it on more.
Common mistake
Trusting one fluent reply, or trusting a judge you have not compared with your own marks. Anthropic's guide says to test LLM grading before you scale it.
Self-check
What is LLM-as-a-judge?
Coming soon. Five questions you already know, marked by you and then by a judge, with the disagreements kept.
Coming soon. How Anthropic separates code grading, human grading, and an LLM judge, and when LangSmith stores a human correction.
Questions
- What is evaluation basics?
- Evaluation basics is checking a model on examples you already know how to grade, including with another model as the judge, before you trust a change.
- Is a judge the same as a fact check?
- No. The judge applies the rubric you gave it. It does not add a source, and Anthropic's guide says to test it before you scale it.
- Should every eval use a model as the judge?
- No. Anthropic's guide says code-based grading is the fastest and most reliable when the task allows it, and to use an LLM grade for judgments that a rule cannot make.