nestonexStart a project How do you know whether a change to an AI feature made it better or worse?
With a fixed set of real inputs and known-good outputs that you run before and after every change. Without one, a model-backed feature has no regression test: prompts, model versions and source documents all shift behaviour silently, and the first report of a problem comes from a customer.
Conventional tests assert exact outputs. Model outputs vary between runs, vary more between model versions, and can change meaningfully because someone improved one line of a prompt. An exact-match assertion is either always failing or pinned to a temperature setting nobody should rely on.
So the unit of testing changes. Instead of asserting a string, you assert a property: that the answer contains the right figure, that it cites a source from the expected document, that it refuses, that it routes to a human, that it stays within a length or a format.
This is the single biggest practical difference between a demo and a shipped feature. A demo works on the examples the builder tried. A feature works on the examples someone wrote down before the build started.
The ordinary cases, drawn from real usage rather than imagination. Twenty to fifty representative requests will surface more than any amount of hypothesising.
The refusals. Questions the system should decline — outside its remit, not covered by its sources, or asking for advice it must not give. These are the cases that quietly break when someone loosens a prompt to fix something else.
The permission boundaries. The same question asked by two users with different access, asserting that each sees only what they should.
The known-hard cases. Ambiguous phrasings, near-duplicate documents, contradictory sources, and anything that has failed in production before. A failure that made it to a customer once belongs in the set forever.
And the adversarial ones. If the feature reads untrusted text and can act, the set needs inputs that try to make it act wrongly.
It has to run on every change that could affect behaviour: prompt edits, model version changes, retrieval configuration, and updates to source documents. Any of those can move the output; the last one surprises people most.
Scoring mixes automated checks with human review. Exact figures, required citations, refusals and formats can be asserted mechanically. Whether an answer is actually good needs a person, on a sample, periodically — and that is worth budgeting for rather than pretending an automated proxy covers it.
Results need to be comparable over time, which means recording them rather than eyeballing them. The question that matters is not 'did this pass' but 'is this better or worse than last week', and you cannot answer that from memory.
The set is a living artefact. Every production failure becomes a case. A set that has not grown in three months is not being used.
The client, and it should be written into the engagement that way. An evaluation set is the most durable thing produced by an AI project: it outlives the prompt, usually outlives the model, and often outlives the vendor.
It is also the thing that makes a future switch possible. Changing provider without an evaluation set is a leap of faith; with one, it is a measurable comparison against criteria you already agreed.
Which is why we treat it as a deliverable rather than a working artefact, handed over with the runbooks and the prompts. If a supplier keeps the eval set, they have kept the only way you had of checking their work.
ProTrader's AI coach is built on exactly this discipline. It is fed each member's own computed broker metrics and wrapped in guardrails that refuse trade recommendations, with a daily spend cap per member. Every one of those is a testable property: does it refuse, does it stay grounded in the member's numbers, does it stop at the cap.
The pattern generalises past AI. What makes a model-backed feature operable is the same thing that makes any system operable — an agreed definition of correct, checked automatically, on every change.
Start at twenty to fifty cases covering the ordinary requests, the refusals and the permission boundaries. Size matters less than coverage: fifty cases spanning the real failure modes catch more than five hundred variations of the same happy path.
Partly. Required figures, citations, formats and refusals can be asserted mechanically, and that covers most regressions. Whether an answer is genuinely useful needs human review on a sample. Using a second model as judge helps at scale but inherits its own blind spots, so it supplements review rather than replacing it.
On every change that can move behaviour: prompt edits, model version updates, retrieval changes and source-document updates. The last is the one teams forget, and it is the one that changes answers without anyone deploying anything.
The same idea under another name — a curated set of inputs with agreed correct outputs, treated as the reference. Some teams distinguish a small hand-verified golden set from a larger evaluation corpus; the distinction only matters if you are maintaining both.