nestonexHow to evaluate an AI feature before you ship it
An AI feature that works in a demo has been tested against the examples its builder happened to try. Evaluation is the practice of writing down, before the build, what it must get right — and then checking that on every change, including the changes nobody deployed.
Evaluate against a fixed set of real inputs with agreed correct outputs, covering the ordinary requests, the ones that must be refused, the permission boundaries and the cases that have failed before. Run it on every prompt, model or source-document change, and record the results so you can see direction rather than a single snapshot.
Why conventional tests do not cover it
Normal tests assert exact outputs. Model outputs vary between runs, vary more between model versions, and shift when someone improves a single line of a prompt. An exact-match assertion on a generated string is either permanently failing or pinned to settings nobody should depend on.
So the unit of assertion changes. Instead of checking a string, you check a property: that the answer contains the correct figure, cites a source from the expected document, refuses, escalates to a person, or stays within a required format. Those are stable across runs and still catch the failures that matter.
This is the practical line between a demo and a feature. A demo behaves on the examples somebody tried. A feature behaves on the examples somebody wrote down before the work began.
- Assert properties, not exact text
- Include the cases that must fail, not only the ones that must pass
- Write the set before the build, not after the first complaint
What belongs in the evaluation set
Start with real usage rather than imagination. Twenty to fifty genuine requests will surface more than a day of hypothesising, because the phrasings people actually use are stranger than the ones you invent.
Then add the categories that quietly break. Refusals: questions the system should decline because they are outside its remit, unsupported by its sources, or asking for advice it must not give. Permission boundaries: the same question asked by two users with different access, asserting each sees only what they should. Known-hard cases: ambiguous phrasings, near-duplicate documents, contradictory sources.
And add every production failure, permanently. A mistake that reached a customer once has earned its place in the set forever, because that is the only mechanism that stops it happening twice.
- Twenty to fifty representative real requests
- Refusal cases and escalation paths
- The same query under two permission levels
- Every failure that has already reached a customer
The regressions nobody deployed
Teams run evaluations after a prompt change and after a model upgrade, which covers two of the three ways behaviour moves. The third is source material: edit a document the retrieval layer reads, and the answers change without anyone shipping code.
That is the failure mode most likely to reach a customer, because it does not look like a release. Nobody runs a test suite after updating a policy page. If your AI feature reads a corpus that people edit, the evaluation has to run on a schedule as well as on deployment.
Provider-side changes are the fourth. Models are updated behind a stable name, and behaviour shifts without any action on your side. Pinning a version where the provider allows it buys predictability; where it does not, scheduled evaluation is what tells you something moved.
Scoring it without pretending
Mechanical checks cover more than people expect: required figures, the presence of a citation, whether the cited document is the right one, refusal behaviour, output format, and length. Those should be automated and run on every change.
Whether an answer is actually good needs a person, on a sample, periodically. Budget for it rather than assuming an automated proxy covers it. Using a second model as judge helps at scale and inherits its own blind spots, so it supplements human review rather than replacing it.
Record results over time. The useful question is not whether today's run passed but whether it is better or worse than last month's, and that is unanswerable from memory. A spreadsheet is enough; the discipline matters more than the tooling.
Who should own the set
The client, and it belongs in the engagement as a deliverable. An evaluation set outlives the prompt, usually outlives the model, and frequently outlives the supplier who built the feature.
It is also what makes a future provider switch a measurement rather than a leap of faith. Changing models without one is hoping; with one, it is a comparison against criteria that were agreed in advance.
If a supplier keeps the evaluation set, they have kept the only means you had of checking their work. That is worth noticing before signing rather than afterwards.
Questions people also ask
How big should an AI evaluation set be?
Start with twenty to fifty cases spanning ordinary requests, refusals and permission boundaries. Coverage beats volume: fifty cases across the real failure modes catch far more than five hundred variations of the same happy path.
How often should the evaluation run?
On every prompt edit, model version change and retrieval configuration change, plus on a schedule to catch source-document edits and provider-side model updates. The scheduled run is the one that catches regressions nobody deployed.
Can a second model score the first?
At scale, usefully. It shares blind spots with the model it is judging and can be influenced by the same inputs, so it supplements periodic human review on a sample rather than replacing it. Keep deterministic checks underneath for anything measurable.
What if we have no real usage to draw from yet?
Write the cases from the workflow the feature replaces: the questions people currently ask a colleague, the documents they currently open. Then replace them with real ones as soon as you have traffic, because the phrasings people actually use are consistently stranger than the ones teams invent.