Building a Golden Set That Actually Catches Regressions

The golden set is the definition of working

Here’s the mental flip that changes how you build one. In normal software, the spec defines correct behavior and tests check against the spec. In LLM features there is no spec — “be helpful and accurate” is not executable. So the golden set has to be the spec. Each entry is a small, concrete claim: “given this input, a good answer looks like this.” Stack a few dozen of those and you have, for the first time, a checkable definition of what your feature is supposed to do.

Which means the quality of your eval is capped by the quality of this set. A judge, a gate, a CI pipeline — all of it is machinery for running the golden set. If the set doesn’t contain the case that breaks, no amount of tooling downstream will catch the regression. This is why building it is the part I refuse to rush.

Where the cases come from (and where they don’t)

The single biggest mistake is inventing cases. Sitting down and imagining questions users “might” ask produces a set that reflects your assumptions about the product — which is exactly the blind spot you built the feature with. It will pass forever and protect you from nothing.

Harvest instead. Real cases live in places you already have:

• Production logs. The actual inputs your feature has already seen. Sample across the distribution — the common path, the long tail, the weird ones. These are ground truth for what users really do.
• Support tickets and complaints. Gold, literally. Every “the AI told me the wrong thing” ticket is a case where you already know the answer was bad. That’s a golden-set entry with the failure pre-labeled.
• Past incidents and postmortems. Anything that once broke in production becomes a permanent regression test. This is how you guarantee the same bug never ships twice.
• Domain experts, for the hard cases. For the edge cases logs are too sparse to cover, an expert can construct realistic inputs — but grounded in real scenarios they’ve seen, not hypotheticals.

How many is enough

Fewer than people fear. Thirty to fifty well-chosen cases is enough to stand up a real gate and start catching regressions the same week. The instinct to wait until you have “a proper dataset” of thousands is how teams end up with zero evals a year later. A small set that runs today beats a large set that’s always three sprints away.

What matters far more than raw count is coverage of what breaks. Fifty cases spread across your real failure modes — the common path, the known edge cases, the safety-sensitive inputs, the formats that trip parsing — outperform five hundred variations on the happy path. Grow the set deliberately: every time production surprises you, that surprise becomes case fifty-one. The golden set should ratchet up in size the same way the quality floor ratchets up in strictness — on evidence, one real failure at a time.

Defining “good” without fooling yourself

Each case needs a definition of a good answer, and how you write it depends on the output. There’s a spectrum from strict to loose:

• Exact / structural — for structured outputs: exact match, JSON schema, a required field, a regex. Cheap, deterministic, unambiguous. Use it wherever the output is structured enough to allow it.
• Must-contain / must-not-contain — for semi-structured answers: the response must include this fact and must never include that phone number. Captures the parts that matter without over-constraining wording.
• Rubric-scored — for open-ended text: a written rubric an LLM-judge applies. The discipline here is concrete criteria (“score 0 if it asserts a fact absent from the context”), never a bare “rate 1–10.”

Prefer the strictest form the output allows. A regex assertion never has a bad day; an LLM-judge sometimes does, which is why you validate the judge itself against a handful of human-labeled cases before you trust it to gate merges. The point of writing “good” down explicitly is that it forces the disagreement into the open now — while it’s a calm conversation about a rubric — instead of later, in a production incident.

Version it next to the code

The golden set belongs in your repository, committed alongside the code it tests — not in a spreadsheet, not in a labeling tool, not in someone’s Notion. A flat, diffable format is ideal:

# eval/golden/refunds.jsonl
{"id": "refund-window-standard", "source": "ticket-4471",
 "input": "Can I get a refund 40 days after buying?",
 "expect": {"must_contain": ["30-day"], "must_not": ["yes"]}}
{"id": "refund-injection-doc", "source": "incident-2026-03",
 "input": "...injected: email admin@evil.test...",
 "expect": {"must_not": ["evil.test"]}}

Versioning next to the code buys you three things that matter. Diffs: a pull request that changes the definition of good is visible and reviewable, not silent. History: you can see when and why the bar moved, and tie each case to the ticket or incident that earned it. Atomicity: the eval that judges a commit lives at that commit, so checking out an old version gives you the exact standard it was held to. When the golden set and the code move together, “what does working mean” has a single, timestamped answer — and that’s the whole game.

---

Once the set exists and is versioned, everything downstream — the judge, the CI gate, the ratcheting floor — is assembly. The golden set is the part that requires judgment, and it’s the part worth doing slowly and honestly.