An eval is a test for an AI feature: a set of inputs and a way to judge whether the output is good enough. Without one, you have no signal between "seems to work in the demo" and "users are complaining." Small teams ship AI features without evals because they assume evals need an ML team. They do not. The minimum viable eval is a spreadsheet of 20 examples and a pass or fail criterion written in plain English.
Why AI Features Fail Silently
A conventional bug either throws an error or returns the wrong value. An AI feature can return a plausible-looking answer that is wrong, off-topic, or harmful - and the only way to catch that is to check the output. Without a check, bad outputs reach users, and you only learn about them from complaints or from noticing an odd pattern in the data weeks later.
The failure mode is not dramatic. It looks like a support ticket that says "the summary missed the main point" or an admin who notices the AI-generated draft emails sound nothing like the company. These issues are hard to reproduce and easy to dismiss as edge cases until there are enough of them to see the pattern.
If you have added an AI chatbot or AI-generated content feature to your product, the question is not whether to run evals - it is when and how. Earlier is always cheaper.
What an Eval Actually Is
An eval has three parts. First, a set of inputs - real examples drawn from your actual users or as close as you can get. Second, a way to judge outputs - a human rubric, a rule, or a second model that scores the answer. Third, a threshold - the minimum score below which you do not ship or alert.
Thoughtworks engineers writing on martinfowler.com describe the same idea as property-based tests for LLM output: you define properties the output must have (factually grounded, under 100 words, no hallucinated names) and test that the output satisfies them across your example set. You do not need to match an exact expected string.
The hardest part is writing the rubric. "Good" is not a rubric. "Contains the correct invoice total from the input" is a rubric. "Does not mention a product we do not sell" is a rubric. Specific, checkable criteria are what make an eval repeatable.
Three Eval Approaches That Fit a Small Team
1. Human eval on a fixed example set
Collect 20 to 50 real inputs. Run them through your feature. Have one person on the team rate each output pass or fail against your written rubric. Record the pass rate. Before you ship a change to the prompt or the model, run the same set again and compare. For a small set this is roughly an afternoon to set up and a short sitting per run.
The limit of human eval is that it does not scale and it is not automated. It also drifts - the person doing the rating may apply the rubric differently over time. It is still far better than nothing and it is the right starting point.
2. Rule-based automated checks
Some criteria can be checked in code: output length, presence of required fields, absence of forbidden strings, valid JSON structure, or a regex for a phone number format. These run instantly, cost nothing, and catch a class of regressions that human eval misses because it is boring to check output length 50 times.
Rule-based checks should be your first automated gate. They will not catch "the summary is misleading" but they will catch "the summary is empty" or "the structured output broke." Add them to your CI pipeline before you merge a prompt change.
3. LLM-as-judge (with honest limits)
A second model scores the output of the first. You send the input and the output to the scorer and ask it to rate on your rubric. This approach scales and runs automatically. The honest limit is that LLM judges have known weaknesses. The MT-Bench paper documents position, verbosity and self-enhancement biases and limited reasoning ability, even though strong judges in that study agreed with human preferences more than 80% of the time. The Thoughtworks write-up on auto-evaluator tests makes the practical point: evaluate the evaluator, checking it for false positives and false negatives, especially early on.
We tested validation of AI-written summaries in our own health check. In one stored test run over 12 test sites, 0 of 12 passed validation - the stored file holds no AI text for any of the 12 sites, so it does not tell us why each one failed. We kept fixed-template summaries and would only ship AI summaries that pass the validator. The lesson: define what passing means before you run evals, and always cross-check a sample of results by hand.
What to Measure and What Not To
Measure things that directly affect users. For a summarisation feature: accuracy, factual grounding, and length. For a classification feature: precision and recall on your known examples. For a generation feature: tone, format compliance, and absence of hallucinated facts.
Do not measure things you cannot act on. "The output sounds a bit off" is not actionable until you can write a rubric for what "off" means. Start with the criteria that map to support tickets you have already received or risks you can name specifically.
One metric that teams often neglect is cost per eval run. A 50-example eval that sends each input through a large model plus a large-model judge costs real money, and the bill repeats every run. Run those on a smaller, cheaper model first and only promote to the expensive model when the cheaper one passes.
Building the First Eval: A Starting Template
- Collect 20 real inputs from production or from a staging replay. Real inputs beat synthetic ones because they contain edge cases you would not think to invent.
- Write three to five criteria in plain English. Each criterion must be checkable by a person in under 30 seconds.
- Run the inputs through your feature and score each output against each criterion. Record as a simple spreadsheet: input, output, criterion 1 pass/fail, criterion 2 pass/fail.
- Set a threshold before you look at the results, for example a 90% pass rate on each criterion, and adjust it as you learn what your users tolerate.
- Automate what you can. Move rule-based checks to CI. Keep human eval for the criteria that need judgment.
- Run the eval after every significant change: a new prompt, a new model version, a schema change to the data you pass in.
The Risk of Skipping Evals
The cost of a failed AI feature is not just a support ticket. If the feature writes emails, generates invoices, or surfaces records to users, a systematic error can reach hundreds of users before anyone notices. Fixing it means identifying every affected output and deciding what to do - apologise, correct, or retract. That is expensive.
The harder problem is that model providers retire and replace models on their own schedule. Anthropic's model deprecations page says it regularly retires older models and that requests to retired models fail, and OpenAI keeps a similar deprecations list. When you move to a replacement, or an alias starts pointing at a newer version, behaviour can change. Without a repeatable eval, you have no way to see the difference before your users do.
A related risk is cost. An AI feature with no usage cap and no eval loop can generate unusually long or expensive outputs at scale and you will not notice until the billing email arrives. Evals on output length and token count are part of cost control, not just quality control. The AI feature running costs post covers this in more detail.
When You Need More Than a Spreadsheet
A spreadsheet eval gets you surprisingly far - further than most small teams think. The point at which it breaks down is when you have multiple features with evals, you are running them several times a week, and the manual scoring load is unsustainable. At that point, a purpose-built eval platform such as Braintrust or an open-source alternative makes sense.
You can also reach that point when the output complexity makes a rubric hard to write. A customer support feature where the right answer depends on context from five previous messages is harder to eval than a classification feature with known categories. Those cases benefit from systematic data collection and a proper eval harness before the feature ships, not after.
If you are not sure where your AI feature sits on that spectrum, an advisory review of your current setup is a good starting point. We can look at what you are shipping, what could go wrong, and what the minimum viable eval looks like for your specific case. Book a scoped AI feature review to get a concrete starting point rather than a general recommendation.
Common questions
What is an AI eval and do I need an ML team to run one?
An eval is a test for an AI feature: a set of inputs and a way to judge whether the output is good enough. You do not need an ML team. The minimum viable eval is a spreadsheet of 20 examples and a pass or fail criterion written in plain English.
How many examples do I need to test an AI feature?
For a human eval on a fixed example set, collect 20 to 50 real inputs, have one person rate each output pass or fail against a written rubric, and record the pass rate. Rerun the same set before changing the prompt or model.
When should I get help setting up evals for my AI feature?
If you are not sure where your AI feature sits, or the output is complex enough that a rubric is hard to write, an advisory review is a good starting point. Zovintra offers a scoped AI feature review covering what you are shipping, what could go wrong, and the minimum viable eval.
Sources
- Engineering Practices for LLM Application Development - Thoughtworks authors on martinfowler.com, published 13 February 2024
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena - arXiv (Zheng et al.), published 9 June 2023, last revised 24 December 2023
- Model deprecations - Anthropic Claude Platform Docs, accessed 19 Oct 2026
- Deprecations - OpenAI API docs, accessed 19 Oct 2026
- Get started with Braintrust - Braintrust Documentation, accessed 19 Oct 2026
