Most teams building with language models test by trying a few examples and deciding it looks better. That works until the first time a change fixes the case in front of them and breaks three they did not look at.
An AI evaluation set is a fixed collection of inputs with agreed correct outcomes, run whenever something changes. It is the single most useful piece of infrastructure in any language model project, and it is usually built after the first incident rather than before.
Start from real inputs
Synthetic examples written by the team tend to be cleaner than reality and skewed toward the cases people already thought about. Real inputs from production, or from the historical process being automated, are far more representative.
Pull a sample of a few hundred and read them. That reading is itself valuable, because it shows the actual distribution of cases rather than the one the project plan assumed.
Choose cases deliberately
- The common cases, in proportion, so the set reflects everyday performance.
- The known hard cases: ambiguous, incomplete, unusual formats.
- Cases where the correct answer is to decline, escalate or say the information is missing.
- Every production failure you have fixed, added as it is fixed.
The third group is where systems most often fail silently. A set containing only answerable cases will never catch a model that confidently answers questions it should not.
The evaluation set is the only honest description of what the system is supposed to do. Without it, everyone is carrying a slightly different version in their head.
Hannah Berg, Lead AI Engineer, Engineered With AI
Agree what correct means
For structured tasks such as classification or extraction, correct is usually unambiguous and can be scored automatically. For generated text it is not, and the scoring approach needs deciding before anyone argues about results.
Break open-ended outputs into checkable properties: does it contain the required facts, does it avoid prohibited claims, is it within length, does it cite the right source. Each property can often be checked by code, which is faster and more consistent than a person reading every output.
Have a person label the ground truth
Someone who knows the domain should decide the correct answers, not the engineer who built the system. Where two experienced people disagree on a case, that case is valuable: it shows where the task itself is ambiguous.
Keep it small enough to run often
A set that takes an hour to run gets run weekly. One that takes a few minutes gets run on every change, which is where its value lies. A few hundred well-chosen cases is usually enough to detect meaningful regressions.
Track the score over time and record which prompt, model and configuration produced each result. A falling score with no recorded change is a sign that something upstream moved.
Read the failures, not just the score
An overall number hides the pattern. A change that improves the average while breaking a whole category is a bad change, and only reading the individual failures reveals it.
Group failures by type and look for the largest group first. It is usually a single underlying problem, and fixing it moves the score more than any amount of prompt tweaking.
Maintain it like code
Store the set in version control, review changes to it, and never quietly edit a case to make a failure disappear. Where a case turns out to have the wrong expected answer, correct it deliberately and note why.
This discipline sits alongside prompt versioning, and the two together are what make changes to a live system safe.
How big should it be?
Statistical rigour matters less than coverage at first. A hundred cases spread across every category you care about will catch far more regressions than a thousand cases drawn from one dominant category.
Grow it where the failures are. Each time production surprises you, add that example along with two or three variants, so the gap closes permanently rather than being patched for a single wording.
Separate a tuning set from a holdout
Teams that adjust prompts while staring at the same examples eventually overfit to them. Keep a portion aside that nobody looks at while iterating, and use it only to confirm that a change generalises beyond the cases it was designed around.
Changing prompts by feel?
We will build an evaluation set from your real cases so every change can be measured.