Build a Quality Ratchet That Blocks Regressions

A quality ratchet is a CI check that remembers your last good score and fails the build if a new one comes in lower. It turns quality into a one-way door. The number can go up or stay flat, but never down. This guide builds one with a small Python script, a checked-in baseline file, and GitHub Actions. Budget about thirty minutes.

#Before you start

• A repo with GitHub Actions enabled.
• A command that already produces a single quality number, such as coverage percent, a lint count, or an eval score.
• Python 3.9 or newer for the ratchet script.
• Write access to commit a baseline file to the repo.

#Store the current score as the baseline

The ratchet needs a floor to compare against. Commit a small JSON file holding your current score. Check it into git so the floor is versioned and moves only through a real commit.

{
  "metric": "coverage",
  "score": 82.4,
  "higher_is_better": true
}

#Write the ratchet check

This script reads the new score, compares it to the baseline, and exits non-zero if it got worse. A non-zero exit is what fails the CI job. It also bumps the baseline in place when the score improves, so the last good score becomes the new floor.

import json, sys

BASELINE = "quality-baseline.json"

def main(new_score: float) -> int:
    with open(BASELINE) as f:
        base = json.load(f)

    old = base["score"]
    higher_better = base["higher_is_better"]
    regressed = new_score < old if higher_better else new_score > old

    if regressed:
        print(f"RATCHET FAIL: {new_score} is worse than floor {old}")
        return 1

    if new_score != old:
        base["score"] = new_score
        with open(BASELINE, "w") as f:
            json.dump(base, f, indent=2)
        print(f"Ratchet raised: {old} -> {new_score}")
    else:
        print(f"Ratchet held at {old}")
    return 0

if __name__ == "__main__":
    sys.exit(main(float(sys.argv[1])))

#Run it locally

Feed it a score below the floor and confirm it fails. Then feed it a score above the floor and confirm it passes and rewrites the baseline. That is the whole behavior, so test both directions before trusting it in CI.

python ratchet.py 80.1   # -> RATCHET FAIL, exit 1
python ratchet.py 85.0   # -> Ratchet raised: 82.4 -> 85.0, exit 0

#Wire it into GitHub Actions

Run your real quality command, capture the number, and pass it to the ratchet. Here the score is set by hand, but any command that prints a number works. The job fails automatically when the script exits non-zero. The workflow runs on pull requests and on pushes to main, and it grants write access so the raise step in the next step can commit.

name: quality-ratchet
on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: write

jobs:
  ratchet:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Produce score
        run: |
          # replace with your real command that emits a number
          echo "SCORE=82.4" >> "$GITHUB_ENV"
      - name: Ratchet check
        run: python ratchet.py "$SCORE"

#Decide how the floor gets raised

On a pull request the improved score updates the file in the runner, but nothing commits it. The simplest honest setup is to raise the floor only on pushes to your main branch, so it moves after code actually merges. Pull requests just get checked against the current floor. The step below runs only on main, which is why the workflow above also triggers on push to main.

# add to the job, after the ratchet step
      - name: Commit raised baseline
        if: github.ref == 'refs/heads/main'
        run: |
          git config user.name  "ratchet-bot"
          git config user.email "ratchet@users.noreply.github.com"
          git add quality-baseline.json
          git commit -m "chore: raise quality floor" || echo "no change"
          git push

#Make it a required check

In branch-protection settings, mark the quality-ratchet job as a required status check for merging. Now a pull request that drops the score cannot merge until someone fixes it. Keep the baseline commit easy to override on purpose. Sometimes a metric legitimately drops, and you need to lower the floor by hand in a reviewed commit.

#Watch out for

• A noisy metric will fail builds for no real reason. If your score wobbles by half a point between identical runs, add a small tolerance to the comparison or the ratchet becomes a coin flip.
• Committing the baseline from CI needs contents: write permission. Scope the commit to main and guard it so an unchanged file does not error.
• A ratchet only guards the one number you feed it. A green ratchet on coverage says nothing about whether the tests are any good. Pick a metric that actually means something, or you are just protecting a vanity number.

#What you built

You have a check that stores your last good score, blocks any pull request that comes in worse, and raises its own floor when quality improves on main. It is a few lines of Python and one workflow, and it makes backsliding a deliberate choice. Point it at a metric you trust, then add a second ratchet for a different dimension once the first one earns its keep.