How many cases are enough?

Fifty gives you a direction; a hundred to two hundred is solid enough to decide on for most projects. Beyond that, more only helps if case variety grows — five hundred cases of one kind teach less than a hundred of different kinds.

Is it reliable to have a model score another model?

Yes, if the criteria are written down and the judge is regularly compared against human scores. Asking 'is this answer good?' with no criteria is not reliable — judge models systematically reward long, polite answers.

How long does building one take?

Usually a day for the first version. The work is not collecting the cases but writing the correct answers, which is done alongside someone who knows the domain. It is often the highest-return day of the project.

← All notes

Measurement

How to build an evaluation set

An evaluation set is the collection of cases that tells you whether your system behaves: an input, an expected output, and a written definition of what counts as correct. It is the only real difference between a demo and a product. It starts at fifty cases and grows for the life of the project.

Published: 2026-09-04
Built by: Zero-Shot Team
3 min read

Why a test set is unavoidable

In ordinary software the same input gives the same output, and a break turns a test red. Language models offer no such guarantee. A word changes in a prompt, a provider updates a model, a temperature setting drifts — and the system quietly starts behaving differently.

The evaluation set is what breaks that silence. Without one, a discussion about quality becomes a discussion about impressions, with no referee between 'this felt better' and 'this felt worse'.

Where the cases come from

The worst source is imagination. A team inventing questions produces the ones the system finds easy; real users always ask differently.

Good sources, in order: support tickets, search logs, first-week sessions with real users, and the cases the system already gets wrong. That last group is the most valuable — every fixed failure is added to the set permanently and never comes back.

Balance matters too. Easy cases, edge cases and cases that should be refused all belong in the set. A set made only of easy cases always scores well and teaches nothing.

Writing down what counts as correct

The hard part is not collecting cases but defining the word 'correct'. For a system producing free text there is no single right answer, and word-for-word comparison does not work.

Three methods are used in practice. Hard checks: is the required fact present, does the citation point at the right document, is the format valid. Rule checks: was a forbidden phrase used, is there an invented number. And a judge model: scoring the answer against hand-written criteria with another model.

All three run together, and the judge itself is checked against human scores on a sample. An unaudited judge produces something that looks like measurement but is not.

When it runs

The set runs every time the prompt is touched, whenever the model version changes, and before every release. Fifty to a hundred cases take minutes, so running it on every change costs almost nothing.

Results are read as a breakdown, not a single number: which category dropped. A stable overall score hiding degraded edge cases is common, and only the breakdown reveals it.

Who owns it

The evaluation set is part of the delivery and stays with the client. The reason is practical rather than principled: whoever maintains the system next needs the tool that proves it still works.

It is also the definition of done. 'The system is ready' says nothing measurable; 'it clears this threshold on this set' does.

In short

Start with fifty cases drawn from real usage, not invented ones.
Add every fixed failure to the set permanently.
Write down what correct means: hard checks, rule checks and an audited judge model.
Run it on every prompt and model change, and read the breakdown, not the total.
The set stays with the client — it is what 'done' means.

Common questions

How many cases are enough?: Fifty gives you a direction; a hundred to two hundred is solid enough to decide on for most projects. Beyond that, more only helps if case variety grows — five hundred cases of one kind teach less than a hundred of different kinds.
Is it reliable to have a model score another model?: Yes, if the criteria are written down and the judge is regularly compared against human scores. Asking 'is this answer good?' with no criteria is not reliable — judge models systematically reward long, polite answers.
How long does building one take?: Usually a day for the first version. The work is not collecting the cases but writing the correct answers, which is done alongside someone who knows the domain. It is often the highest-return day of the project.

More notes

All notes →