# Evaluation basics

> Evaluation basics is checking a model on examples you already know how to grade, including with another model as the judge, before you trust a change.

Checking a model on examples you already know the answer to, before you trust a change.

Track: Foundations
Access: free
Card: https://buildsparkai.com/evaluation-basics
Plain text: https://buildsparkai.com/evaluation-basics.md
Last checked: 2026-10-18

## What it is

An eval is a set of examples you can grade. Anthropic's guide to success criteria and evaluations says a successful LLM application starts by defining what success is, then designing evaluations that measure performance against it. The same guide says to automate grading when you can, with code or with an LLM, and that more graded questions beat a handful of replies you only read by hand. Code-based grading is the fastest and most reliable when a rule can decide. Human grading is the most flexible, and the guide calls it slow and expensive. LLM-based grading is the option for a judgment a rule cannot make, after you have tested that the judge is reliable. That second model is what this card calls LLM-as-a-judge.

## Why it matters

**Cost.** Each example is a model request, and a judge is another request on the reply. Anthropic's guide says human grading is slow and expensive, and that an LLM grade is the scalable option after you have checked it. This card does not add a price.

**Quality.** A change that looks better on one reply can be worse on the set. Anthropic says to design evals that mirror the real task, including edge cases, so you can see whether a prompt change helped before users see it.

**Risk.** A judge can pass a reply a person would fail, or the reverse. Anthropic says to test LLM grading for reliability before you scale it. LangSmith's guide describes collecting human corrections so the judge stays closer to those marks. A passing eval is not proof the reply is true outside the examples.

## Try today

Write five questions you already know the answer to. Run your prompt. Mark each reply pass or fail yourself. If you use a model as the judge, compare its marks with yours on those five before you trust it on more.

## Common mistake

Trusting one fluent reply, or trusting a judge you have not compared with your own marks. Anthropic's guide says to test LLM grading before you scale it.

## Self-check

What is LLM-as-a-judge?

Another model grades the reply against a rubric or a known answer

## Practice and deep

Coming soon. Five questions you already know, marked by you and then by a judge, with the disagreements kept.

Coming soon. How Anthropic separates code grading, human grading, and an LLM judge, and when LangSmith stores a human correction.

## Sources

- [Anthropic, Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)
- [LangSmith, LLM-as-a-judge evaluator](https://docs.langchain.com/langsmith/llm-as-judge)

Opened 2026-10-18. Vendor and framework docs. This card does not add a rate, a usage count, or a score from a leaderboard.