Glossary

Plain-English definitions — no jargon for its own sake.

Golden set — a curated collection of real inputs paired with agreed-good outputs, used as the source of truth for scoring an AI feature.
LLM-as-judge — using an evaluation model to score another model’s output against criteria like faithfulness, relevance, and safety.
Faithfulness / grounding — whether an answer is actually supported by its source material, rather than invented.
Hallucination — a confident, fluent answer that is not true or not grounded in any source.
Prompt injection — an input crafted to make the model ignore its instructions and do something else.
Jailbreak — an attempt to bypass a model’s safety rules, often via role-play or a fake “no-rules” persona.
CI quality gate — an automated check in your pipeline that blocks a merge or deploy when a quality score drops below a floor.
Ratchet — a gate whose passing floor only ever moves up, so quality can’t silently erode.
Flake — a test that passes and fails without any code change; flaky suites train teams to ignore red.
Golden run / evidence — a saved, reproducible test run (traces, screenshots) that proves a result.
Human-in-the-loop — a design where a person approves an AI action at a defined risk point.
RAG — retrieval-augmented generation: grounding answers in retrieved documents instead of model memory.