Wire an Eval Gate into a GitHub or Vercel Deploy Pipeline

You will make your eval run inside a deploy pipeline so a failing eval stops the deploy. This covers both paths: a required GitHub Actions check that blocks the merge, and a Vercel build command that fails the deployment when the eval fails. Budget about twenty minutes if you already have an eval that exits non-zero on failure.

#Before you start

• An eval you can run with one command that exits non-zero when it fails (Promptfoo, a script, anything).
• A repo connected to GitHub, and to Vercel if you use the Vercel path.
• Your model API key ready to add as a CI secret, never in the repo.

#Make the eval a single command with an honest exit code

The whole gate depends on one thing: the command must exit non-zero when the eval fails. A pipeline reads the exit code. It does not read your report. Add a script so both GitHub and Vercel call the same entry point. Verify it locally by checking the exit code after a run.

{
  "scripts": {
    "eval": "promptfoo eval -c promptfooconfig.yaml"
  }
}

#Confirm the exit code before you trust it

Run the command and print the exit code. If a failing eval prints red text but still exits 0, no gate will ever fire. This is the one check people skip, and it is why gates silently pass. Make the failing case actually fail here first.

npm run eval
echo "exit code: $?"   # must be non-zero when a case fails

#Gate a merge with GitHub Actions

Add a workflow that runs the eval on every pull request. A non-zero exit fails the job, and that failing job is what you will make required. Put the API key in repo secrets under Settings, Secrets and variables, Actions.

name: eval-gate
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

#Make the check required

A job that runs but does not block is just a report with extra steps. In the repo settings, add a branch protection rule for your main branch and mark eval-gate as a required status check. Now a pull request that fails the eval cannot be merged until someone fixes it or updates the cases on purpose.

#Gate the Vercel deploy itself

If you deploy on Vercel and want the eval to block the actual build, run it inside the build command. If npm run eval exits non-zero, the whole build fails and nothing ships. Add your model key as an environment variable in the Vercel project settings so it is available at build time.

{
  "buildCommand": "npm run eval && next build"
}

#Watch out for

• The eval runs on every deploy in the Vercel path, so a slow or expensive suite slows every build. Keep the build-time suite small and run the full one as a required GitHub check instead.
• A required check that people routinely make optional under deadline pressure is theatre. If it gets bypassed a lot, the cases are probably wrong. Fix them.
• LLM-graded evals are non-deterministic and can flap red on a good change. Keep rubrics tight, and do not set the pass bar so high that normal variation fails the build.

#What you built

You now have an eval that can actually stop a bad change, either at the merge as a required GitHub check or at the Vercel build itself. The gate is only as honest as the exit code and the cases behind it, so keep both real. Next, grow the case set from the failures production hands you, especially the ones that embarrassed you.