Your LLM feature needs a regression suite more than a better prompt
Prompt tweaks feel like progress because nobody is measuring. Golden traces + LLM-as-judge in CI turn "it seems better" into a diff you can gate on.
Every team I talk to is tuning prompts. Almost none of them can answer the only question that matters: did this change make the feature better or worse? Without measurement, prompt engineering is redecorating in the dark.
The fix is the same discipline we’ve had for code for twenty years, adapted for non-determinism. A golden set: 50–200 real inputs with agreed-good outputs, versioned next to the code. A judge: LLM-as-judge scoring faithfulness, relevance, and safety against that set — imperfect, but consistent enough to catch regressions. A gate: the suite runs in CI, and a score drop below the ratcheted floor blocks the merge.
In nexural-qa-os this became ten dedicated runners — hallucination, jailbreak, prompt-injection, toxicity, PII-leak, refusal, bias, consistency, cost, latency — each producing a computed score from a real command, never a stated one. The scorecard is reproducible: run it twice, get identical output.
Start smaller than you think: even 30 golden traces and one faithfulness judge in CI beats zero. The floor ratchets up from there — every improvement becomes the new minimum, and quality stops silently regressing while everyone stares at the prompt.