Examples
Test a prompt change on cases you already know the answers to
You'll go from "the new prompt seems better" to knowing it fixed your problem and broke nothing that worked.
Example: made up to show the idea.
You're ready if you've done the basics, ending with Evaluation basics. No code yet? You can run the cases by hand in a Chat assistant.
Tool to reach for: an AI model through your own developer account (an API: the way your code sends text to a model and gets a reply). Nothing here connects to us.
Steps
- Write 10 made-up questions your support assistant should answer, each with the known answer. No real customer messages.
- Make 2 of them questions your documents don't answer. The known answer is "I don't know".
- Write a pass rule for each case. Use an exact match where a rule can decide.
- Run your current prompt on all 10, and record pass or fail.
- Change one thing in the prompt. Run all 10 again.
- Keep the change only if it passes the check below. Save the cases for next time.
No code? Run it by hand
Open a new chat for each case, so earlier cases don't shape the reply (Lesson 1). Paste your prompt, then the question. Copy each reply into a table, and mark it with your pass rule.
One row of your results table, filled in
- Case. 7: "Can I change the delivery address after my order ships?" (your documents don't cover it)
- Expected answer. "I don't know", and no rule made up
- Old prompt. "Yes, within 24 hours of shipping."
- New prompt. "I don't know. Please contact the support team."
- Pass or fail. Before: fail. After: pass.
The change was one added line: "If the documents don't answer the question, say 'I don't know'." Cases 1 to 6 and 8 to 10 must still pass after it.
Prompt to copy, for cases a rule can't grade
You are grading one reply. Known answer: [known answer]. Reply: [reply]. The reply passes if it states the known answer and adds no other fact. First give one sentence of reasoning. Then give one mark: PASS or FAIL.
What good output looks like
A table of 10 cases, with a before and an after mark for each. The case you set out to fix flips to pass, and no other case changes.
Check before you trust it
Watch for: a fix that breaks a case that used to pass.
Compare the before and after marks case by case. Before you trust the grader prompt, run it on 5 cases you graded by hand.
- Pass: the target case now passes, every case that passed before still passes, and the grader agrees with your hand marks.
- Fail: any case flips from pass to fail, or the grader disagrees with you. Fix the prompt or the grading rule, then run all 10 again.
What goes wrong
You test only the case that made you change the prompt. It passes, and a different question quietly starts failing.
Learn the idea
Try it on your own task
Before your next prompt change, write 10 cases you know the answers to, and run them before and after.
More whole tasks, simple to advanced →
Where the tool facts come from: Evaluation basics and Lesson 1, each with its sources. Last checked 2026-10-18.