LLM regression testing with Promptfoo in CI: the minimum viable gate

30 golden traces, one judge, one failing exit code. The smallest setup that stops a bad prompt change from reaching production — runnable this afternoon.

Every LLM team I talk to has the same confession: prompt changes ship because they "seemed better" in a couple of manual checks. Nobody can prove the last change didn’t make something else worse, because nothing is measuring. Here is the smallest setup that fixes that — not a platform, not a framework migration. One config file, one CI step, one afternoon.

fig. 1 — the minimum viable gate: four stops, no exceptions

Step 1 — the golden set. Collect 30 real inputs from logs or support tickets — not invented ones; real phrasing is weirder than anything you’ll write. For each, record the output your domain expert agrees is good. Commit them next to the code. This file is now the definition of "working," and changing it requires a reviewed diff — which is the entire point.

Step 2 — the judge. A minimal promptfooconfig.yaml that runs every golden input through your prompt and scores the output with an LLM rubric:

prompts:
  - file://prompts/support-answer.txt
providers:
  - anthropic:claude-sonnet-5
tests: file://golden/*.yaml   # 30 cases: vars + assertions
defaultTest:
  assert:
    - type: llm-rubric
      value: >-
        Faithful to the provided context, answers the actual
        question, no invented policies or prices.
    - type: cost
      threshold: 0.02

Step 3 — the gate. Promptfoo exits non-zero when assertions fail, so CI needs exactly one honest step:

# .github/workflows/eval.yml
- run: npx promptfoo eval --config promptfooconfig.yaml
  # non-zero exit = merge blocked. that's the whole gate.
```
Wire it to trigger on changes to <span style="font-family:'JetBrains Mono',monospace;font-size:0.9em">prompts/**</span>, <span style="font-family:'JetBrains Mono',monospace;font-size:0.9em">golden/**</span>, and wherever your model or retrieval config lives. A red eval that can’t block a merge is a report, not a gate.

Your first eval suite doesn’t need to be good. It needs to exist, run in CI, and be allowed to say no.

What I add when teams outgrow the minimum — in rough order of payoff: a ratcheting floor (every improvement becomes the new minimum, so quality can’t silently regress); safety runners for injection, PII, and toxicity on the same golden set; retrieval-quality metrics in front of generation for RAG; and per-request cost budgets that fail the run like any other assertion. That fuller battery is what my LLM evaluation engagement builds — but the 30-trace version above is free, takes an afternoon, and catches the regression you currently can’t see. Start there.