Ragas: Automated Evaluation of Retrieval Augmented Generation

A RAG system can fail in several ways. The retriever can return the wrong passages, or the right ones buried in noise. The generator can ignore the passages and answer from memory, or answer a slightly different question. A single "was the answer good?" score cannot tell these apart, and in 2023 most teams had nothing better, partly because good evaluation seemed to need human-written reference answers they did not have.

Ragas, from Exploding Gradients and Cardiff University, proposes three metrics that need no reference answer, only the question, the retrieved context and the generated answer.

• Faithfulness: is every claim in the answer supported by the context? A model breaks the answer into short statements, then checks each against the context. The score is the fraction supported.
• Answer relevance: does the answer address the question? A model writes questions that the answer would answer; the score is how similar those questions are to the real one, measured with embeddings.
• Context relevance: is the retrieved context focused? A model extracts the sentences needed to answer; the score is the share of the context they make up.

To test the metrics, the authors built WikiEval: 50 recent Wikipedia pages, questions about them, and pairs of answers or contexts that two human annotators compared. Ragas agreed with the humans 95% of the time on faithfulness, 78% on answer relevance and 70% on context relevance, well ahead of simply asking a model for a score.

When a customer says "the assistant gives wrong answers", Ragas-style metrics turn the complaint into a location.

Low faithfulness means the generator is the problem. The answer says things the context does not. Fix the prompt, tell the model to say when the context is insufficient, or change models. Retrieval was not the issue.

Low context relevance means retrieval is noisy. The right passage may be there, surrounded by irrelevant ones. Fewer, better-ranked passages usually help, both for cost and because models use focused context better.

Low answer relevance means it answered the wrong question, or half of it, or padded it. Often a prompt or query-rewriting issue.

Two cautions come from reading the paper closely. First, its test set contrasted easy cases, such as an answer written with the source against one written without it. Real failures are subtler, so treat 95% as an upper bound and calibrate on the customer's data. Second, nothing in the three metrics checks whether the retrieved context contained the needed information at all. An answer can be perfectly faithful to an incomplete context. Measuring that, retrieval recall, needs labeled questions, so keep a small labeled set alongside the reference-free metrics.