We built a free app health check tool that audits websites against 26-28 checks covering security headers, performance signals, SEO basics, and common configuration errors. The tool generates a summary for each site. At some point the obvious question became: could an LLM write a better summary than our hand-written templates? We set up a test. In one stored snapshot of that test, 0 of 12 runs produced an AI summary that passed our validation, and the file shows why that number is weaker than it sounds: no AI summary was generated at all. We are writing this post because the validation problem is more useful than the headline number.
Note: this post describes a specific internal experiment stored in our codebase (web/gemini-eval.json). The test sites are fictional. The results are from a single snapshot. We are not claiming this is a generalised finding about LLMs - it reflects our validation criteria and our test set. We are sharing it because we think the failure modes are instructive.
What the Experiment Was
The health check tool runs a series of checks on a given URL. Each check produces a finding: pass, fail, or warn, with a short explanation. The template-based summary reads those findings and produces a paragraph like: 'We verified 26 of 26 checks from public information. The main items: no Content Security Policy is set; browsers are not told to always use HTTPS.'
The template summaries are reliable but formulaic. We wanted to know whether an LLM could produce a more readable, prioritised, client-friendly version of the same findings. The test was designed to send the structured findings to Gemini and ask it to summarise them.
The validation criteria we designed cover several properties: factual grounding (the summary should only mention issues that appeared in the findings), appropriate severity framing (a CSP warning should not be described as a critical security breach), absence of hallucinated recommendations (the LLM should not suggest fixing issues that were not found), and reading level appropriate for a non-technical site owner.
What Failed and Why
The stored result in our codebase shows 0 of 12 passing. Looking at the stored data, the aiSummary field is null for all 12 rows, the aiUsed flag is false, and the token counts are zero. So the file records no AI generation in this snapshot, and the 0 of 12 reflects that, not a judgement of Gemini's writing.
What we can say with confidence: none of the 12 runs produced an AI summary that passed validation, because no AI summary text was stored in this snapshot. The failure modes below are typical LLM risks a validator should catch, not findings from this file:
- Summaries that described findings more severely than the data supported - a missing optional header described as 'critical vulnerability'
- Generic recommendations not grounded in the actual findings - 'consider adding two-factor authentication' when authentication was not tested
- Inconsistent severity framing across the 12 sites - the same CSP-missing finding described differently depending on what else was present
- Summaries that omitted the most important finding in favour of a more dramatic-sounding lesser one
The template summaries, by contrast, scored consistently on factual grounding because they are generated deterministically from the findings data. They cannot hallucinate a recommendation that does not follow from the checks.
The Validation Problem Is Harder Than the Generation Problem
The more interesting lesson from this experiment is that writing the validator was harder than generating the summaries. To validate an AI-generated audit summary, you need to:
- Define precisely what counts as a factual claim
- Have a machine-readable representation of the ground truth (the findings)
- Write logic to check each claim in the summary against the ground truth
- Handle paraphrase - 'no HTTPS redirect' and 'HTTP requests are not redirected to HTTPS' mean the same thing
- Decide what an acceptable reading ease score is for your target audience
This is the same problem that makes LLM output validation hard in general. The output looks like natural language, but your validator needs to treat it like structured data. Building a validator that is reliable is almost as much work as building the thing you are validating.
If you are building a product that uses an LLM to generate reports, summaries, or recommendations, and you do not have a validator, you have no signal on quality. You are shipping unknown output to real users. The fact that an LLM output reads well is not evidence that it is correct. Our guide to evals for small teams explains how to build this kind of check without a machine learning background.
What This Means for AI-Generated Reports
A common pattern in AI-built products is that report generation is added as a feature without an evaluation framework. The LLM produces output that looks authoritative - professional language, good structure, reasonable recommendations. The founder or developer reads a few outputs, they seem fine, and the feature ships.
The problem surfaces later, when a user follows a recommendation that was hallucinated, or when a severity label is wrong and causes unnecessary alarm. By then the feature has been live for weeks and the trust damage is done.
Our current position: template-based summaries for the health check tool. They are less readable than a well-prompted LLM output, but they are grounded and consistent. When we do add LLM summaries, they will go through the same validator and will not ship until they pass.
What We Would Do Differently
If we ran this experiment again, we would first confirm that the generation step actually runs and stores its output, and then change three things. First, provide the LLM with the findings in a more structured prompt format - explicitly listing which checks passed and which failed, with severity labels, before asking for a summary. Second, constrain the output - ask for a summary of no more than three sentences, each grounded in a specific finding. Third, run the validator iteratively during prompt development, not only at the end.
None of those changes guarantee a passing result. They reduce the search space the LLM has to work within, which tends to reduce the rate of unsupported claims. But there is no prompt that eliminates the need for a validator.
The honest conclusion from this experiment is that AI-generated reports need the same quality infrastructure as any other software output: defined criteria, automated checks, and a human review layer for the cases the automation misses. If you are adding AI report generation to your product and are not sure where to start, run the free health check and then talk to us about what a review of your AI feature would involve.
Common questions
Can an LLM write reliable website audit summaries?
In Zovintra's single stored snapshot, 0 of 12 runs produced an AI summary that passed validation, but no AI summary was generated at all, so it is not a judgement of the model. The experiment is not a general finding about LLMs.
Why is validating AI-generated reports harder than generating them?
A validator must define what counts as a factual claim, hold machine-readable ground truth, check each claim against it and handle paraphrase. Building a reliable validator is almost as much work as building the thing being validated.
How should I check an AI feature that generates reports for users?
Define criteria, automate checks and keep a human review layer for what automation misses. No prompt eliminates the need for a validator. Zovintra offers a review of what an AI feature would involve, after you run the free health check.
Sources
- web/gemini-eval.json - Zovintra internal evaluation file (12 fictional test sites), accessed 19 Oct 2026. No external studies are cited in this post.
