The eval method

#The four moving parts

1. A golden set — real inputs paired with agreed-good outputs, versioned next to the code. This is the source of truth.
2. A judge — an evaluation model (or deterministic check) that scores each output for faithfulness, relevance, and safety.
3. Safety probes — adversarial inputs that try to break the feature the way a real user or attacker would.
4. A CI gate — the suite runs on every change; a score below the ratcheted floor blocks the merge.

#The principles

• Computed, not claimed. A score comes from a command anyone can re-run — never from an opinion.
• Ratchet, don’t just pass. The floor only goes up, so quality can’t silently erode over time.
• Earn down human review. Start with a human in the loop; remove approval points only when evals prove it’s safe.
• No fake green. If the gate should be red, it goes red — publicly. (This site’s own scorecard works the same way.)