RAG Evaluation Frameworks for Grounded Answer Quality
Split retrieval and generation scoring to catch hallucinations end-to-end evaluation misses.
A RAG-powered legal assistant was asked about contract termination notice. It answered with confidence: "Section 4.2 of the contract requires written notice 30 days before termination." The retriever had returned five chunks for that query, and not one of them mentioned Section 4.2. The eval harness scored the answer correct anyway, because it used string matching and the user's question contained the words "termination" and "notice." The system looked healthy from the outside while actively hallucinating a citation that did not exist in the source document.
That case names the structural problem with evaluating RAG systems on a single end-to-end accuracy number. A RAG pipeline has two failure surfaces that operate independently of each other: retrieval, which decides what context the model sees, and generation, which decides what the model does with it. Collapse both into one score and that score cannot tell which surface broke. A model with excellent retrieval can still ignore the context it was handed, invent a citation, or blend a retrieved fact with something from its own parametric memory in a way that reads as authoritative and is wrong. Retrieval quality explains something like three-fifths of the variance in final answer quality, by some estimates, which leaves a substantial share of errors originating downstream of retrieval that a retrieval fix alone will never touch.
Hallucination became the dominant production failure mode for RAG systems largely because end-to-end evaluation cannot see it coming. Retrieved context primes the tone of a model's answer even when the model does not actually use the retrieved facts, so the output sounds grounded whether or not it is. A team running a labeled benchmark and watching its accuracy numbers hold steady might reasonably conclude the system is working. But a correct-looking final answer can come from junk retrieval paired with a hallucinating model, or from perfect retrieval paired with a model that ignored every word of it. Benchmark accuracy, measured as one number at the end of the pipeline, has no way to tell those two situations apart.
The three-layer model: what belongs at each evaluation surface
Grounded answer quality calls for scoring three distinct layers rather than one: retrieval, generation, and end-to-end. Each layer answers a different question, and a passing score at one layer says nothing about what happened at another.
The retrieval layer asks whether the right chunks came back from the corpus. That question gets measured with context precision, context recall, Mean Reciprocal Rank (MRR), and hit-rate at k. These four do not move together. A retriever can rank the single best chunk first and score well on MRR while still missing several other chunks the answer needed, which weakens context recall. Precision, recall, and MRR need to be tracked as separate numbers, because a retriever that looks strong on one can be quietly failing on another, and averaging them away hides exactly the information a developer needs to diagnose the system.
The generation layer asks whether the model used the retrieved context faithfully once it had it. The generation layer is where hallucination actually lives, a layer end-to-end evaluation cannot see: an end-to-end score only tells a team an answer was wrong, leaving the cause buried in whichever layer produced it.
The end-to-end layer asks whether the final answer was correct and useful in the terms the user cares about. That gets measured with answer correctness, helpfulness, refusal calibration, and format compliance. This layer matters and should be tracked, but it only confirms whether a system is working.
The production pattern that makes this three-layer architecture operational is span-attached evaluation. A trace where retrieval scored low and the final answer still scored high means the model filled the gap with something that was not in the retrieved context, which is hallucination by definition, occurring exactly where end-to-end accuracy would have reported a win.
Of the three layers, the generation layer carries the most consequential distinctions for a team trying to catch hallucination before it reaches a user, and the next section stays there.
Faithfulness and groundedness are not synonyms, and conflating them misses real failures
Faithfulness and groundedness get used almost interchangeably in casual conversation about RAG evaluation, and that habit obscures two different failure modes. A system can pass one of these metrics while failing the other, and the gap between them is where some of the costliest hallucinations hide.
Groundedness measures whether a response is supported by the retrieved context: a grounded answer makes no claim that goes beyond what the retrieved documents actually contain. Faithfulness measures something narrower and, in some ways, more demanding: whether the response accurately represents what the source material says, without distorting, omitting, or inverting its meaning. A response can satisfy groundedness and still fail faithfulness. Consider a model summarizing a retrieved study that reported a negative correlation between two variables; a response claiming the study found a positive correlation stays entirely within the bounds of the retrieved context (it is "about" that study) while completely misrepresenting what the study said. Grounded, and wrong.
That gap matters more once scoring moves from the aggregate to the granular. Suppose a five-sentence answer has four sentences solidly supported by retrieved chunks and a fifth sentence that is fabricated. A single faithfulness score computed across the whole response, on a 0 to 1 scale, might average out to something that clears a passing bar, because four correct sentences can outweigh one invented one in a blunt aggregate. Groundedness scored at the sentence level does not allow that kind of averaging to hide the problem: it flags the fifth sentence specifically as ungrounded, which gives a developer the exact location of the failure instead of a vague signal that something, somewhere, in the response might be off.
Context utilization adds a third, complementary signal. An answer can look correct and still report low context utilization, which should read as suspicious rather than reassuring: it suggests the model answered from its own parametric knowledge instead of the retrieved material sitting in front of it, even though the words it produced happen to be right. Faithfulness scoring alone would not catch that case, because the output is accurate. It's groundedness and context utilization together that expose it, because the model's reliance on the provided context, not just the correctness of its final words, is what each of them is actually measuring.
How to set operational thresholds that gate deployments
Once faithfulness and groundedness are tracked as separate numbers, the practical question becomes what to do with each score. Translating a metric into a deployment decision calls for separate thresholds, because the right response to a groundedness failure and the right response to a faithfulness failure are not the same action.
A workable starting point: flag responses that score below 0.80 on groundedness for human review, and block responses that score below 0.70 on faithfulness before they ever reach a user. The asymmetry is deliberate. A response that is grounded but imperfect is a candidate for a reviewer to look at and possibly approve with context; a response that actively misrepresents its source material should not leave the API boundary at all, because an unfaithful claim delivered to a user cannot be recovered after the fact the way a flagged-for-review response can. That is the real distinction driving the different thresholds: which failure a human reviewer can still catch downstream, and which one has already done its damage the moment it reaches a screen.
Regulated domains raise the bar further. In legal, medical, and financial applications, a faithfulness threshold above 0.85 is increasingly treated as a floor rather than a target, because deployment below that line raises hallucination exposure in exactly the domains where a hallucinated citation or a misstated dosage carries the highest cost. None of these numbers function as universal law. The right threshold for a given system depends on the domain, the cost of a wrong answer reaching a user, and how much review capacity a team actually has, and a team should treat 0.70 and 0.80 and 0.85 as starting points to calibrate against its own risk tolerance.
Refusal calibration deserves its own gate, separate from faithfulness. A model that under-refuses, answering confidently when the corpus simply does not contain the answer, creates a safety problem, since a confident wrong answer is often more dangerous than an honest "I don't know." A model that over-refuses, declining to answer questions the corpus actually could have supported, creates a usability problem instead. Both deserve tracking as their own metric rather than folding into the faithfulness number, because fixing one can make the other worse if they are watched as a single blended score.
Two retrieval-stage interventions affect how often these gates fire. Cross-encoder re-ranking, applied after initial retrieval, improves precision by removing irrelevant chunks before they ever reach the model, and enterprise benchmarks tie that improvement in precision to a reduction in hallucination: raise retrieval quality upstream and the faithfulness gate has less to catch downstream. HyDE retrieval, which embeds a hypothetical answer before searching, works in the opposite direction on vague queries: it tends to improve recall, since the hypothetical answer gives the search something more specific to match against, but it can hurt precision when that hypothetical answer itself hallucinates and pulls in the wrong chunks. A threshold system catches that kind of degradation at the output; a monitoring system that attributes the failure to the retrieval stage is what tells a team a retrieval-stage intervention like this, specifically, is the cause.
None of these thresholds do any good sitting in a specification document. They function as gates only if a team has tooling that can score every response against them automatically, at the volume production traffic actually runs, which is a tooling question.
What the major RAG evaluation frameworks score
No single framework covers every evaluation need a RAG system raises, and the right choice depends on whether a team wants an open-source metric library, a CI/CD-integrated gate, a research-grade judge, or an inference-time deployment block.
RAGAS established the pattern most teams encounter first: four metrics, faithfulness, answer relevance, context precision, and context recall, scored with an LLM acting as judge. It is open-source, free, and functions as the canonical starting point for developers who want standard RAG metrics plus synthetic test generation inside a Python workflow.
TruLens, also open-source and free, organizes its evaluation around what it calls the RAG Triad: did retrieval find useful information, is the answer supported by that information, and did the answer address the user's question. TruLens suits teams that prefer thinking in terms of that triad framing, or teams already using Snowflake, since Snowflake maintains TruLens as an open-source project.
LlamaIndex offers its own open-source evaluation tooling built for developers already working inside the LlamaIndex application framework, testing retrieval quality directly in code. It is free.
Two research-track frameworks push past what the production-oriented tools above attempt. ARES, developed at Stanford, generates tailored LLM judges for each RAG component using synthetic training data combined with prediction-powered inference, a statistical technique that lets it use a small number of human-annotated examples to correct for the judge model's own errors. ARES beats RAGAS by substantial margins on context relevance accuracy and answer relevance accuracy, predicts hallucination occurrence rates close to ground truth, and needs far less human annotation than baseline approaches to get there. It also reports confidence intervals around its scores, a property most other frameworks simply do not offer, and it stays accurate across domain shifts, even when the type of query or document changes from what it was built on.
CCRS, developed at Carnegie Mellon in 2025, takes a different architectural approach: a suite of five metrics, Contextual Coherence, Question Relevance, Information Density, Answer Correctness, and Information Recall, scored by a single pretrained LLM acting as a zero-shot, end-to-end judge. By skipping the intermediate claim-extraction and entailment-checking steps that multi-stage pipelines rely on, CCRS runs substantially faster in practice, while still matching or beating RAGChecker's discriminative power on key aspects like recall and faithfulness.
On the production side, Openlayer runs groundedness and faithfulness scoring at inference time itself, with deployment gates built to block unfaithful responses before they ever leave the API boundary, which is the thresholding discussion from the previous section implemented directly as infrastructure rather than left as a policy a team has to enforce by hand.
Domain-specific benchmark sets fill a different gap: general-purpose metrics do not always capture what matters in a specialized corpus. RAGBench, CRAG, LegalBench-RAG, WixQA, and T²-RAGBench each give teams evaluation data suited to a particular domain. NVIDIA publishes its own accuracy matrix across these benchmark sets, using end-to-end RAG answer accuracy computed through RAGAS, with each response rated on a discrete scale of 0, 2, or 4 by an LLM judge.
Matching a framework to a use case starts with the question a team is actually trying to answer. A developer wanting fast, free metrics inside a Python pipeline has RAGAS or TruLens as reasonable starting points. A research team building toward low-annotation, confidence-scored evaluation has ARES and CCRS. A production team needing scoring enforced at the moment a response would otherwise reach a user has tools like Openlayer built specifically for that boundary. None of these replace the layered thinking laid out earlier in this piece: whichever framework a team picks, it only earns its keep if it is scoring retrieval, generation, and end-to-end separately, rather than handing back one number and calling the system healthy.
