Project essay

Building AI evaluations that explain what changed

How I built model-quality measurement for Snap eHealth’s production AI, so a release rests on evidence of what changed rather than a demo that looked good.

Thomas Howard · 7 min read

My role

Evaluation architecture & implementation

Focus

Python · Amazon Bedrock · Model evaluation

Project stage

Live production · Gates real releases

A health chatbot can look excellent in a demo and still be wrong in the way that matters. Change a prompt or swap a model, and a summary could start dropping what a patient said, or turn a denied symptom into a reported one. Fluent text hides that kind of failure well.

Without a way to measure it, every release is a judgment call made on a few good-looking responses. I designed, built, and operate the evaluation system that now gates releases of Snap eHealth’s production AI. When a model, prompt, or policy changes, it tells engineers what changed and where, and gives the person approving the release the evidence to decide.

The central suite covers a patient-facing health chatbot and Health Risk Assessment (HRA) summaries. Controlled synthetic scenarios exercise ordinary behavior, difficult edge cases, and adversarial input.

Write down what “correct” means before you score anything

The first risk is measuring the wrong thing. A chatbot and a summary can both sound fluent while failing at completely different jobs. One generic quality score hides more than it shows.

So I start with the output’s job. The chatbot has to respond appropriately and stay inside its scope. The HRA summary has to preserve the meaning of what the patient actually answered. Both need privacy and safety controls, but each gets its own behavior contract.

The contract is concrete. If the patient denies a symptom, the summary must keep it as a negative finding. Missing information stays missing. Uncertainty stays visible. Those properties are testable, and they still allow many acceptable phrasings.

Goldsets supply the reference cases, with separate collections for guardrails, adversarial instructions, and expanded clinical quality. I report results by category, because a strong average can conceal a consequential failure in a small family of scenarios.

Never let a good score overrule a failed safety check

An LLM judge is good at nuance and bad at being a rule. If its score can outvote a hard check, a well-written unsafe answer passes.

The system uses two kinds of evidence with a strict order between them. Deterministic checks enforce explicit requirements: required behavior, forbidden content, privacy rules, expected concepts. A separate model with its own rubric judges semantic quality such as completeness, clarity, scope, and faithfulness. The judge can accept a reasonable wording difference. It cannot turn a hard failure into a pass.

One run is also not enough. Repeated sampling keeps every generation and its outcome, which shows whether the model is stable or just got lucky once. Paired scenarios hold the substance of an input constant and vary only controlled demographic wording. That separates two different problems: drift across repeated runs, and a change in behavior caused by who the patient appears to be.

The cost is that you now depend on a second model. I keep the judge’s rubric and explanation next to every score, so a surprising judgment can be inspected and calibrated against human review. That matters most when the candidate and the judge share a blind spot.

Make every result traceable to the configuration that produced it

A regression you can’t reproduce turns into an argument. Someone remembers a different prompt, a result gets overwritten, and the team debates instead of fixing.

I treat the evaluation bundle as the unit of change. Prompts, model parameters, guardrail policies, datasets, and evaluation configuration travel together. Fingerprints of the dataset and prompt registry identify exactly what was tested, and a reproducibility manifest records the seed.

For fixture-backed runs, canonical JSON makes identical inputs byte-for-byte reproducible. Real model calls vary, so there the bundle pins the conditions being compared and the individual samples capture the variability. Reports keep the checks, judge explanations, samples, and category and cohort summaries, with sensitive fields redacted before anything is stored. An engineer can follow an aggregate drop down to the exact failing case.

The gate itself is code. Release criteria combine overall pass rate, critical failures, repeated-sample stability, judge quality, and paired-scenario consistency. When the configured gate fails, the runner exits with a failing status. Downstream automation gets a deterministic boundary, not a dashboard to interpret.

Find out who is wrong before you touch the prompt

The most expensive mistake in evaluation is fixing the wrong thing. If every red result triggers a prompt edit, you end up tuning the model to satisfy a broken test.

I sort failures into four classes: the model behaved poorly, the evaluator raised a false positive, the expected answer was ambiguous, or the case is really a policy decision. A model defect calls for a candidate change. A faulty assertion calls for a test fix. An open policy question goes to its owner.

Sally gave me a concrete case. A faster speech path let a conversational test advance before background analysis had arrived, so part of the apparent regression was the test’s own pacing. I made turn timing and the arrival of accepted memory explicit, so the test measured conversational continuity under the intended workload.

The easy fix was to block every turn on analysis. The test would have passed, and the latency design it was meant to protect would have been gone. Every repair keeps the original failure, the changed configuration, and the new evidence together, so an independent reviewer can judge whether the fix addressed the cause.

Test the evaluator, then measure the model

An evaluator is software, and software has bugs. If the measuring stick is wrong, its results look authoritative and still mislead.

The suite has three execution paths: mock, fixture, and real provider. Offline tests cover case loading, rule application, report generation, redaction, and gate behavior. Mock runs give fast feedback whenever the measurement code itself changes.

Real-provider runs use Amazon Bedrock for the candidate and a separate model as judge. The recorded chatbot baseline used a Sonnet candidate with a Haiku judge, and it is retained with its configuration and scenario coverage so later runs have an honest comparison point. The HRA suite stays separate, with reference, guardrail, and expanded cases aimed at the summary contract through expected concepts, faithfulness checks, and semantic review. Separate results per product make a regression easier to locate.

Public Amazon Bedrock evaluation documentation describes the platform’s evaluation options. My contribution is the application-specific part: the contracts, datasets, runners, result handling, and interpretation that make those options useful for Snap’s systems.

A healthy service can still give a bad answer

Ordinary software checks tell you the service responded. They won’t tell you the summary left something out, or that the chatbot crossed a conversational boundary. Behavioral evaluations put those failures next to the ordinary checks.

Release review is organized around the tested bundle and its failure evidence: candidate and judge identities, scenario coverage, repeated samples, and changes by category. That gives the reviewer enough context to decide what progresses, what needs repair, and what needs a policy decision.

The discipline is simple to state. Preserve the regression, fix its cause, and keep it in the suite. Each production lesson becomes part of the next release decision.

What I’d tell a founder

Define “correct” per output before you pick an evaluator. A single quality score across different products will hide the failure you most need to see.

Put a hard order between rules and model judgment. An LLM judge is useful, but it should never be able to pass something a deterministic check failed.

Treat the evaluator as production code. When a result looks like a regression, find out whether the model, the test, or the policy is wrong before anyone edits a prompt.