How I test AI systems when the answer is probabilistic

The unit-test mindset dies on contact with an LLM. The first thing every engineer tries is assert output === expected — and it fails immediately, because the model produced a perfectly good answer worded differently. The wrong conclusion is that AI systems can't be tested. The right conclusion is that you're testing the wrong thing.

Stop testing strings. Test properties.

A probabilistic answer still has deterministic properties. Those are your assertions:

• Grounding — every claim cites a source that was actually in the retrieved set. An answer citing outside it fails, no matter how good it sounds.
• Structure — the output parses, matches the schema, includes required fields. This is a plain old assertion.
• Behavior — the system refuses when it should: off-topic, injection attempts, questions the corpus can't answer.
• Drift — the score on a fixed golden set doesn't fall when the prompt, model, or index changes.

// one eval case — properties, not prose
{
  "id": "refund-policy-04",
  "question": "Can I get a refund after 30 days?",
  "must_cite": ["policies/refunds.md"],
  "must_not": ["hallucinated_source", "definitive_legal_advice"],
  "expected_behavior": "answer_with_caveat",
  "rubric": ["grounded", "complete", "correct_refusal_scope"]
}

The anatomy of the harness

Four parts, all versioned in the repo next to the code they test:

A golden set of question → expected-source pairs, built from real usage and deliberately seeded with traps: questions the corpus can't answer, near-duplicate sources, and adversarial phrasings. A scoring rubric that turns "is this good?" into named, checkable dimensions. Versioned prompts with a changelog — a prompt edit is a code change and gets the same review. A runner that emits a JSON report per run, so results diff cleanly and archive with the release.

Three graders, in order of trust

1. Exact checks — schema validation, citation-set membership, banned-phrase scans. Cheap, deterministic, run on everything. 2. Heuristic rubric scoring — length bounds, required-topic coverage, source counts. Slightly fuzzy, still explainable. 3. LLM-as-judge — useful for nuance like tone and completeness, but a judge is itself a probabilistic system: it gets its own spot-checks against human grades, and it never gets the final word on a release by itself.

Treat every change as a regression risk

Model swap, prompt edit, re-chunked index, new embedding version — each one reruns the full golden set, and the diff of scores is reviewed like a failing CI run. The most dangerous failures are silent: retrieval quietly returning worse sources while answers stay fluent. That's why retrieval gets its own smoke tests, separate from answer quality — you need to know which layer regressed.

Human review is a feature, not a fallback

When the workflow touches users, money, or reputation, low-confidence answers route to an approval queue instead of shipping. That queue is also your best eval-case generator: every human correction is a new golden-set entry. The loop closes — production feeds the tests that guard production.

None of this makes the model deterministic. It makes the system accountable — which is the actual job.