Service
Fine-tuning and evaluation
An evaluation set is a hand-labelled test collection that shows whether an AI system actually works. Zero-Shot builds that set first, then fine-tunes a model on your data if it is still needed. A model without a measure cannot be improved, and any claim of improvement cannot be checked.
What we build
- 01An evaluation set built from real cases with correct answers labelled by hand
- 02An automatic scoring pipeline that every release runs through before deployment
- 03A regression gate that blocks a release when it is worse than the last one
- 04Fine-tuning on your data (LoRA or full) with a before-and-after comparison
- 05A comparison table showing what each prompt, model and parameter change did to the result
How it runs
01
Define success
We write down what a good answer actually means. This step is harder than it sounds and is often the most valuable output of the project.
02
Build the set
Real cases are sampled and labelled by hand. Edge cases and known failures go in alongside the easy ones.
03
Cheap improvements first
Prompt, context and tool changes are measured. In most projects the target is met without fine-tuning anything.
04
Fine-tune if needed
When prompting is not enough we fine-tune and compare against the same set. If there is no gain, the fine-tuned model is not shipped.
What we measure
- Accuracy on the set, and accuracy on edge cases, reported separately
- Regression count: cases that were right in the previous version and are wrong now
- Cost and latency per case
- The gap between the fine-tuned model and the base model on the same set
- How well the evaluation set represents real traffic
When this is the wrong fit
If the team does not agree on what the right answer is, an evaluation set cannot be built — that agreement comes first. And with only a few hundred examples, fine-tuning is not advisable: at that volume prompt and context work is both cheaper and better.
By sector
The same work runs differently in every sector. The detail is here.
Common questions
Fine-tuning, RAG or prompting?
The order is: if knowledge is missing, RAG; if format and tone are missing, fine-tuning; if instruction is missing, prompting. In most projects the problem turns out to be missing knowledge, which makes fine-tuning unnecessary. We decide this by measuring on the same evaluation set, not by guessing.
How many examples do we need?
For an evaluation set, 100-300 well-chosen examples cover most jobs. Fine-tuning needs a few thousand, and their quality matters more than their count. Fine-tuning on too few examples usually makes the model worse.
Who labels the evaluation set?
Your team, because the domain knowledge is yours. We build the set, provide the labelling interface and measure agreement — if two people label the same case differently, that gets resolved before anything is measured.
Other services
What we've written on this
- Where to start with enterprise AIHow to choose a first AI project: which tasks qualify, in what order to proceed, and what to expect in the first eighty days.
- Writing prompts for productionWhy a prompt that works in a demo breaks in production. Treating prompts like code: structure, versioning, boundaries and measuring every change.
- Why AI makes things up, and how to reduce itWhy language models state false things confidently. The real cause of hallucination, which countermeasures work, and which do not.
Last updated: 2026-09-05
Let's talk about this one
Tell us what you want to build and within two days we'll tell you who takes it and how we'd measure it.