Set Up a Nightly LLM Regression Suite in GitHub Actions

A nightly regression suite runs your LLM through a fixed set of cases while nobody is watching, so slow quality drift shows up as a failed run instead of a support ticket. This guide schedules an eval with GitHub Actions and pings you when it fails. It builds on an existing eval config, the same kind Promptfoo uses. Budget about twenty minutes.

#Before you start

• A repo with an eval you can run from one command (this uses promptfoo eval).
• An API key for your model, stored as a repo secret.
• A Slack incoming webhook URL, or another alert you can hit with curl.

#Confirm the eval runs from one command

Before you schedule anything, make sure the suite runs and exits non-zero when a case fails. A scheduled job is only useful if a failure actually fails the process. Run it once locally to be sure.

npx promptfoo@latest eval -c promptfooconfig.yaml
echo "exit code: $?"

#Add the secrets

Put your model API key and the Slack webhook in the repo's Actions secrets, under Settings then Secrets and variables then Actions. Never paste either one into the workflow file. The job reads them from the environment at run time.

#Schedule the workflow

Add a workflow with a schedule trigger. The cron below runs at 07:00 UTC every day, which is nighttime in the US. Keep workflow_dispatch too, so you can trigger a run by hand instead of waiting until tomorrow to test it.

name: nightly-eval
on:
  schedule:
    - cron: "0 7 * * *"
  workflow_dispatch:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Run nightly eval
        run: npx promptfoo@latest eval -c promptfooconfig.yaml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

#Alert when it fails

A failed run in the Actions tab is easy to miss. Add a step that fires only when an earlier step failed, using if: failure(), and posts to Slack. The message links back to the exact run so you can open the logs in one click.

      - name: Alert on failure
        if: failure()
        run: |
          curl -X POST -H 'Content-type: application/json' \
            --data "{\"text\":\"Nightly eval failed: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID\"}" \
            "$SLACK_WEBHOOK_URL"
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

#Test it without waiting for night

Open the Actions tab, pick the nightly-eval workflow, and use Run workflow to trigger it now. Then break a case on purpose, run it again, and confirm the Slack message lands. If the alert does not arrive, you have a broken alarm, which is worse than no alarm.

#Watch out for

• GitHub cron runs in UTC, and scheduled jobs can be delayed by several minutes or skipped entirely when the runners are busy. Nightly is fine for this. Do not use the schedule trigger for anything that needs to fire at an exact minute.
• Scheduled workflows get disabled automatically after 60 days of no repo activity. If your nightly run goes quiet, check whether GitHub paused it and re-enable it.
• A run that always passes might just mean your cases are too easy. If the suite has never failed, feed it a known-bad prompt and confirm it actually catches it.

#What you built

You now have an eval that runs itself every night and messages you the moment quality drops, with a link straight to the logs. It catches the slow drift that no single pull request would trip. Next, widen the case set from real production misses, so the suite keeps testing the world your users actually live in.