Shadow deployment
Shadow deployment means running your new version next to the old one on real production traffic, while only the old version's answers reach users. The new version sees the same live requests and does its work in the dark, and you compare the two. The catch is that it has to be truly invisible. Anything with a side effect, like charging a card or sending an email, needs to be stubbed or it will act for real.
Why it matters
Staging traffic is polite and predictable. Real traffic is weird: the typos, the huge inputs, the questions nobody thought to test. Ship a new model or prompt straight to users and those surprises land on customers first. A shadow run lets the new version fail privately on real inputs, so you catch the crash or the quality drop before anyone downstream feels it.
How it works
You fork each incoming request to both versions, serve only the old one's response, and log both to compare. Then you measure the gap: latency, error rate, and how far the new answers drift from the current ones on the same inputs. Watch for a while, since a change that looks fine at noon can break on the overnight batch or the Monday spike. This pairs with a canary, where the new version starts serving a small slice once the shadow run looks clean.
A support bot is switching to a cheaper model. You shadow it for a week: every real user question hits both models, users only see the old one, and both answers get logged. On day three the logs show the new model quietly botching every refund-window question. You fix it before a single customer ever saw one of those answers.