Answer relevancy
Answer relevancy measures whether the response actually addresses the question the user asked. A relevant answer stays on the exact thing that was requested. An irrelevant one drifts to a related topic or buries the point under padding that never lands on what was asked.
Why it matters
An answer can be perfectly true and still useless if it talks past the question. A user asks "how do I cancel," gets three paragraphs about your pricing tiers, and leaves annoyed. Low relevancy is how a system fails while looking like it works: the text is fluent and factual, just aimed at the wrong target. It pairs with faithfulness, which checks that the answer is grounded in your sources. Faithfulness asks "is it backed up," relevancy asks "is it even on-topic."
How it works
A common automated method flips the answer around. An LLM generates the questions this answer would be a good reply to, then you compare those questions back to the original using embedding similarity. If the answer is on-point, the reconstructed questions look a lot like the real one and the score is high. Wandering answers, or ones stuffed with hedges and filler, produce questions that drift from the original, and the score drops.
A user asks a docs bot, "Does the free plan support webhooks?" The bot replies with an accurate, well-written overview of every plan tier and its price. Every word is correct, but it never says yes or no about webhooks on the free plan, so it scores low on answer relevancy. Right topic, wrong question.