CI quality gate
A CI quality gate is an automated check in your build pipeline that blocks a release when a quality score falls below a set threshold. It runs your evals on every change, and if the number is too low, the merge or deploy stops. The hard part is picking a threshold strict enough to catch real regressions but loose enough that noise alone doesn't fail every run.
Why it matters
Without a gate, evals become a dashboard nobody looks at. Someone ships a prompt tweak, the score quietly drops from 92 to 78, and the first person to notice is a user with a screenshot. A gate makes the pipeline itself refuse to ship the regression, so the machine enforces quality instead of whoever remembers to check. That is the whole no fake green idea: the build is green only when the evidence says it earned it.
How it works
Run your golden set as a step in CI and compare the score against a threshold you commit to the repo, like "faithfulness must stay above 0.85." Pass, and the pipeline continues. Fail, and the build goes red and the PR is blocked. Flaky evals can trip a gate on noise, so people often run each case a few times and gate on the average. A ratchet is the stricter cousin: it only lets the threshold move up, never down.
A team ships a refund support bot behind a gate set at 90 percent on their eval set. An engineer swaps in a cheaper model to save cost, and the eval run drops to 84 because the model starts inventing return windows. The gate fails the pipeline, the deploy never happens, and the engineer sees the broken cases before a single customer does.