The Minimum Viable Eval Gate

Why a gate, not a dashboard

Most teams shipping an LLM feature have a dashboard somewhere — a notebook, a spreadsheet of example outputs, a Slack channel where someone pastes “this looks better now.” None of it blocks anything. A prompt tweak, a model version bump, a retrieval change: they all ship because they seem fine, and the regression they caused surfaces three weeks later in a customer complaint nobody can trace back to the diff.

A gate is different from a dashboard in one way that matters: a gate can say no. It sits in your pull-request pipeline, it runs on every change that touches the AI, and when the quality score drops below a floor you agreed on, the merge is red. That’s the whole idea. You don’t need a platform, a vendor contract, or a data-labeling team to get there. You need thirty examples, one scoring rule, and a CI job.

The three parts

Strip an eval system down to its irreducible core and you get three things. Everything else — safety runners, RAG retrieval metrics, cost budgets, ratcheting floors — is elaboration on top of these.

1. The golden set

A small collection of real inputs paired with what a good answer looks like. Thirty to fifty is enough to start — pulled from your actual logs and support tickets, not invented at your desk. This set is your definition of “working,” so it has to reflect the cases you actually care about: the common path, the edge cases that bite, and the ones a customer already complained about. (There’s a whole guide on building this well; the short version is: harvest, don’t imagine.)

2. The judge

Something that scores each output. For structured outputs, that’s deterministic assertions — exact match, regex, JSON-schema validation, a required substring. For open-ended text, it’s an LLM-as-judge: a second model given the input, the response, and a rubric, asked to score faithfulness, relevance, or safety on a fixed scale. The trick that makes a judge trustworthy is a written rubric with concrete pass/fail criteria — not “rate this 1–10,” but “score 0 if the answer states a fact not supported by the provided context.”

3. The gate

A CI step that runs the golden set through the judge, aggregates the scores, and compares the result to a floor. Above the floor: exit 0, the merge is allowed. Below: exit non-zero, the check is red, the merge is blocked. It runs on every pull request that touches a prompt, a model config, a retrieval index, or the code around them.

The afternoon build

Here is the honest sequence. None of these steps is research; they’re assembly.

• Hour 1 — harvest. Pull 30–50 real inputs from production logs or your support queue. For each, write down (or agree on) what a good answer contains. Store it as a flat file — JSONL or YAML — in the repo, next to the code it tests.
• Hour 2 — the judge. Write the rubric. For each output, decide whether it’s a deterministic check (assertion) or an open-ended one (LLM-as-judge). Keep the rubric in version control too — it changes as your definition of good changes, and you want that history.
• Hour 3 — the runner. A script that loops the golden set through your feature, scores each output, and prints a summary. Open-source runners like Promptfoo or DeepEval give you this loop, assertion types, and an LLM-judge harness out of the box, so you write config, not plumbing.
• Hour 4 — the gate. Wire the runner into CI as a required check on pull requests. Fail the job when the aggregate score is below the floor. That’s the line that turns a script into a gate.

The CI step is smaller than people expect. Conceptually:

# .github/workflows/eval-gate.yml (shape, not gospel)
on: pull_request
jobs:
  eval-gate:
    steps:
      - run: npm ci
      - run: npx promptfoo eval -c eval/gate.yaml --output out.json
      - run: node eval/check-floor.js out.json --floor 0.85
        # check-floor.js exits 1 if the aggregate pass-rate < 0.85

Mark the job a required status check on your protected branch, and the platform does the enforcement for you: a red gate means the merge button is disabled. No human has to remember to look.

Setting the floor without lying to yourself

The first floor is easy to get wrong in two opposite ways. Set it too high and every honest PR is red, so the team learns to bypass the check — a gate everyone routes around is worse than no gate, because it launders bad changes with a green badge. Set it too low and nothing ever fails, so you’ve built a dashboard with extra steps.

The move that works: run the suite on your current main to get today’s baseline score, then set the floor at that baseline. Now the gate’s only job on day one is “don’t get worse than we already are.” That’s a claim you can defend and a bar every real improvement clears. Over time you ratchet: when a change legitimately raises the score and holds, you raise the floor to match. The floor only moves up, and it moves on evidence.

One more discipline: when the gate blocks a PR, the failure output has to be readable by the person who wrote the diff — which traces failed, what the judge said, and the delta from baseline. A gate that just says “score 0.81, floor 0.85, blocked” gets resented. A gate that says “these 3 of 40 traces regressed on faithfulness, here’s what each one said” gets trusted, and trusted gates survive.

---

That’s the minimum viable version. The elaboration — a safety probe battery, RAG retrieval metrics, per-run cost budgets, ratcheting floors that climb automatically — is the next section of the same system. But none of it matters until the three parts above exist and the gate can turn red. Build that first.