Canary prompt
A canary prompt is one fixed input you send through your AI system over and over, on a schedule or on every deploy, because you already know what a good answer looks like. The name comes from the canary in a coal mine: when it stops singing, something is wrong. The moment the answer drifts, you hear about it before your users do.
Why it matters
Models and vendors change under you without warning. A provider quietly ships a new model version, a config flag flips, or a retrieval index goes stale, and a task that worked yesterday starts returning garbage. A canary prompt turns that invisible slide into a loud alarm. You find out from a failing check at 9am instead of an angry customer at 9pm.
How it works
Pick a few inputs that exercise the paths you care about most, and pin the expected output or a simple pass rule for each. Run them on a cron and after every deploy. Compare each fresh answer to the known-good one with exact match, a similarity threshold, or a strict check on the one field that matters. When a canary fails, alert and hold the release. Keep the set small and stable so a red result means a real regression, not a flaky test.
A support bot has one canary: "What is your refund window?" with the pinned answer of 14 days. Every hour a job asks that exact question and checks the reply. The provider rolls out a new model minor version overnight, the bot starts saying 30 days, and the canary goes red at 2am. You catch it from the alert, before a customer tries to return something on day 20.