Champion-challenger
Champion-challenger keeps the model that currently works in production, the champion, in charge of real traffic. A new candidate, the challenger, has to earn its spot by beating the champion on the same tests before it gets promoted. The trap is promoting on a gut feeling or one good demo instead of a measured win.
Why it matters
New models and prompts feel like upgrades and often are not. Swap in a challenger too early and you can quietly regress on cases that were working fine, and you only hear about it from users. Keeping a champion means production never gets worse than a known-good baseline, and every change has to prove itself before it touches anyone.
How it works
Run the challenger against the same golden set the champion is scored on and compare the numbers side by side. Promote only when the challenger clears the champion by a margin wide enough to rule out noise. Many teams go further and route a slice of live traffic to the challenger in the background, a shadow deployment, so they can watch how it handles today's real inputs before flipping the switch. Once promoted, the old champion becomes the new baseline to beat.
Your support bot runs on last quarter's model. A newer model lands and looks sharper in a few chats, so switching is tempting. You run it against your 80-case golden set instead. It wins on refunds but drops two account-recovery cases the champion nailed. The champion stays in charge until the challenger fixes those and comes back ahead.