Reference
Eval gates: the missing CI step for AI →
The cornerstone: what an eval gate is, its anatomy, and why AI features need one.
04 · Pillar
AI Testing in CI/CD
Treat AI quality like code quality: gate it. Wiring evaluation into the pipeline — eval gates, quality ratchets, regression suites, canary releases, and "no fake green" as an actual workflow.
Field note
LLM regression testing with Promptfoo in CI →
The minimum viable eval gate you can add to CI this week.
Field note
Your LLM feature needs a regression suite →
Why a regression suite beats a better prompt, every time.
Guide
Human-approval checkpoints, explained →
Where a human belongs in an otherwise-automated AI pipeline.
Reference
AI under test →
What it looks like when an AI system is held to a real testing standard.
Tutorial
Build an LLM Eval Gate in CI with Promptfoo →
Add a Promptfoo eval gate to CI that blocks a merge when quality drops.
Tutorial
Add a Human-Approval Checkpoint to an AI Pipeline →
Insert a human-in-the-loop approval step before risky AI actions.
Tutorial
Set Up a Nightly LLM Regression Suite in GitHub Actions →
Run an LLM eval suite nightly in GitHub Actions with alerts.
Tutorial
Add Open-Source LLM Observability (Langfuse or Phoenix) →
Add self-hosted LLM observability with Langfuse or Phoenix.
Tutorial
Build a Quality Ratchet That Blocks Regressions →
Build a ratchet where the last good score becomes the new floor.
Tutorial
Wire an Eval Gate into a GitHub or Vercel Deploy Pipeline →
Make an eval gate a real deploy blocker in GitHub or Vercel.
Guide
Shift-Left Testing for AI: Catch Regressions Before Merge →
How to move AI quality checks earlier, to the pull request.
Guide
Canary Releases for AI Models: A Practical Playbook →
A practical playbook for canary-releasing AI models and prompts.
Guide
Setting SLOs for AI Features: What Good Enough Means →
How to set service-level objectives for AI features.
Guide
The AI Incident Post-Mortem, With a Real Example →
A post-mortem template for AI incidents, with a worked example.
Continue across the library
Need this on your product, not just in theory?
Book a 15-minute call and we'll scope the smallest useful next step.