The point of evaluating RAG in two layers is that retrieval and generation fail for different reasons, and a single score cannot tell you which one broke 1. This recipe builds the smallest useful two-layer eval. retrieval metrics (context precision and context recall) on a known answer set, plus generation metrics (faithfulness to retrieved context, answer relevancy) using Ragas as the metric library 2.
Ragas is a Python library that wraps the standard RAG metrics over an LLM judge. It is not a tracing platform; it computes scores you can pipe into your observability hub. We use it because the metric implementations are the literature reference implementations and because the API is short 3.
The dataset shape
You need four columns per row:
question. The user query.contexts. The chunks your retriever returned for the query.answer. The response your RAG pipeline produced.reference. The gold answer, read by both retrieval metrics.
Ragas itself names these user_input, retrieved_contexts, response, and reference. Its Dataset path still accepts the shorter names used here and maps them for you.
Without a gold-answer column you can still run faithfulness and answer relevancy. Context precision and context recall both read reference. If you have no gold answers, Ragas ships a response-based precision variant, ContextUtilization, which this recipe does not use 4.
For a thirty-question dev set with a small corpus, a CSV is fine. At scale, JSONL with one query per line scales better.
question,contexts,answer,reference
"What is the company's parental-leave policy?","Eligible employees receive 16 weeks of paid leave...||Paid leave can be taken in two blocks within 12 months...","Eligible employees get 16 weeks of paid parental leave, which can be split into two blocks taken within 12 months of birth or adoption.","Eligible employees receive 16 weeks paid parental leave, which can be taken in up to two blocks within 12 months of birth or adoption."
We use || to delimit multiple retrieved chunks in a single CSV cell. The loader splits on it.
Figure: Which dataset columns each Ragas RAG metric requires in version 0.4.3: all four read the question, context precision and context recall also require the gold reference answer, faithfulness checks the generated answer against the retrieved contexts, and answer relevancy compares the answer to the question. Skipping the reference column still lets you run two of the four metrics.
Step 1: install and load
pip install "ragas==0.4.3" "langchain-community<0.4" langchain-openai openai pandas datasets
export OPENAI_API_KEY=...
Pin both. 0.4.3 is the latest Ragas release on PyPI as of 2026-10-01 (published 2026-01-13). An unpinned install resolves langchain-community 0.4, which no longer ships the vertexai module that Ragas 0.4.3 imports at startup, so import ragas raises ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai' (reproduced in a clean Python 3.12 venv on 2026-10-01).
"""Two-layer RAG eval using Ragas.
Run:
pip install "ragas==0.4.3" "langchain-community<0.4" langchain-openai openai pandas datasets
export OPENAI_API_KEY=...
python ragas_eval.py
"""
from __future__ import annotations
import os
from pathlib import Path
import pandas as pd
from datasets import Dataset
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas import evaluate
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.metrics import (
context_precision,
context_recall,
faithfulness,
answer_relevancy,
)
def load(path: str) -> Dataset:
df = pd.read_csv(path)
# contexts is a list of strings per row in Ragas.
df["contexts"] = df["contexts"].apply(
lambda s: [c.strip() for c in s.split("||")]
)
# Ragas maps these names to user_input, retrieved_contexts, response, reference.
return Dataset.from_pandas(df)
def main() -> None:
ds = load("rag_eval.csv")
print(f"Loaded {len(ds)} rows.")
judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-2024-08-06", temperature=0))
emb = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-small"))
result = evaluate(
ds,
metrics=[
context_precision,
context_recall,
faithfulness,
answer_relevancy,
],
llm=judge,
embeddings=emb,
)
print(result)
# to_pandas() returns Ragas's column names; map them back so the CSV
# keeps the question/contexts/answer headers the later scripts read.
out = result.to_pandas().rename(columns={
"user_input": "question",
"retrieved_contexts": "contexts",
"response": "answer",
})
out_path = Path("ragas_results.csv")
out.to_csv(out_path, index=False)
print(f"\nWrote per-row scores to {out_path}.")
print("\nMean scores:")
for m in ("context_precision", "context_recall", "faithfulness", "answer_relevancy"):
if m in out.columns:
print(f" {m:22s} {out[m].mean():.3f}")
if __name__ == "__main__":
main()
Run it:
python ragas_eval.py
You will see Ragas iterate over each row, call the judge model for each metric, and print a per-metric mean at the end.
Ragas 0.4 prints deprecation warnings for four things in this script. The metric imports from ragas.metrics are slated for removal in v1.0; evaluate(), LangchainLLMWrapper, and LangchainEmbeddingsWrapper are marked for removal in a future version. The docs' replacement is the collections API: metric classes from ragas.metrics.collections built on llm_factory, scored one row at a time with await metric.ascore(user_input=..., response=..., retrieved_contexts=...) 3. This recipe stays on evaluate() because it returns the per-row frame the later steps read, and the version pin keeps it reproducible. Port it before you move past 0.4.
What the four metrics mean
Each metric is a different question about a different layer.
Context precision asks: are the relevant chunks ranked at the top of what the retriever returned? It averages precision@k over the ranks that hold a relevant chunk, so a noisy chunk ahead of the relevant ones costs more than one behind them 4. The judge marks each chunk relevant or not against the reference answer, one call per chunk. If precision is low, your retrieval is over-pulling or ranking badly 1.
Context recall asks: of the claims in the gold answer, what fraction are supported by the retrieved chunks? High recall means the retrieval did not miss any of the needed information. Low recall means your retriever under-pulled (or the chunks do not cover the question). Like context precision, it needs the reference column 2.
Faithfulness asks: of the claims in the generated answer, what fraction are supported by the retrieved context? Low faithfulness means the model hallucinated beyond what the retrieval gave it. Ragas implements this with claim-extraction followed by per-claim verification, the same pattern as FActScore 3 5.
Answer relevancy asks: does the generated answer actually address the question? It is the cheapest sanity check; if it drops while faithfulness holds, the answer is grounded in the retrieved text but does not address the question, which usually means the retriever pulled the wrong document.
The four split cleanly into two layers:
flowchart LR
subgraph Retrieval
A[context_precision]
B[context_recall]
end
subgraph Generation
C[faithfulness]
D[answer_relevancy]
end
A --> C
B --> C
Step 2: diagnose failures by layer
When you read the per-row CSV, the pattern of low scores tells you what to fix.
| Pattern | Layer to fix |
|---|
| Low context_precision, normal everything else | Retriever returns noisy chunks. Raise the relevance threshold or rerank. |
| Low context_recall, normal everything else | Retriever misses chunks. Lower the threshold, expand chunk size, or add a hybrid search. |
| Normal retrieval, low faithfulness | Generator hallucinates. Add explicit "cite the chunk" prompt instructions and constrain to retrieved context. |
| Normal retrieval, low answer_relevancy | Generator drifts off-topic. Check the prompt template; the model may be over-paraphrasing. |
| Everything low | Either the retrieval index is broken or the question set is impossible for this corpus. |
The mistake teams make is reporting a single "RAG quality" number. Aggregating across layers hides the diagnostic signal that makes the metrics actionable 1.
Step 3: report per-row, not just means
Ragas returns per-row scores. Read them. The mean is for the dashboard; the per-row scores are where you find the bug.
import pandas as pd
df = pd.read_csv("ragas_results.csv")
# Faithfulness disasters (likely hallucination).
hallucinations = df[df["faithfulness"] < 0.5].sort_values("faithfulness")
print(f"Found {len(hallucinations)} likely hallucinations.")
hallucinations[["question", "answer", "faithfulness"]].to_csv(
"hallucinations.csv", index=False
)
# Retrieval misses (context_recall low).
misses = df[df["context_recall"] < 0.5].sort_values("context_recall")
print(f"Found {len(misses)} retrieval misses.")
misses[["question", "reference", "context_recall"]].to_csv(
"retrieval_misses.csv", index=False
)
You read the hallucination CSV by hand. The pattern almost always clusters into one of three buckets: (a) the model invented a number not in the context, (b) the model combined two chunks into a claim neither chunk supports, or (c) the model paraphrased loosely enough that the literal claim is unsupported even though the gist is right. Each bucket has a different fix.
Step 4: a sanity gate before you trust the numbers
Run faithfulness on a handful of trivially-correct rows (the gold answer exactly equals one of the retrieved chunks) to verify the judge is not over-flagging. Then run it on a few trivially-wrong rows (an answer that contradicts the chunks) to verify the judge is not under-flagging.
If either sanity check fails, fix the rubric on the judge before you trust the headline numbers. The Ragas judge prompt is in their repo if you need to override it 3.
Step 5: wire it into CI
Pin a fixed dev set. Run Ragas on every PR that touches the retriever, the chunker, or the prompt. Fail the PR if context_recall or faithfulness drops more than five points from the main-branch baseline:
import json
import sys
import pandas as pd
BASELINE = json.load(open("baseline_metrics.json"))
TOLERANCE = 0.05
df = pd.read_csv("ragas_results.csv")
metrics = {
"context_precision": df["context_precision"].mean(),
"context_recall": df["context_recall"].mean(),
"faithfulness": df["faithfulness"].mean(),
"answer_relevancy": df["answer_relevancy"].mean(),
}
failed = []
for k, v in metrics.items():
base = BASELINE[k]
if v < base - TOLERANCE:
failed.append((k, v, base))
if failed:
print("REGRESSION:")
for name, got, base in failed:
print(f" {name}: {got:.3f} < {base:.3f} - {TOLERANCE}")
sys.exit(1)
print("OK")
For the GitHub Actions wrapping, see production CI integration. For trace logging the retrieved chunks at production time, see Langfuse online eval.
What this skips
This recipe does not cover synthetic query generation for the eval set itself; for that, see synthetic query generation. It does not cover long-context RAG (RULER-style multi-needle); see long-context eval with RULER. It does not cover non-English RAG, which behaves materially differently because both the embedding model and the judge have language-specific biases. The OpenAI cookbook RAG eval notebook is a good companion to this one 6. If you have labeled relevant documents per query, the evals FAQ's advice is to score retrieval with plain IR metrics (recall@k, precision@k, MRR) and spend judge budget on generation, where error analysis and targeted judges do the work 7; see retrieval metrics.
CAUTION
Ragas calls the judge more than once per metric per row: faithfulness makes two calls per row and context precision makes one per retrieved chunk. A 200-row dev set retrieving five chunks per query is about 1,800 judge calls, not 800. Pin a small dev set for CI and run the full set nightly. Cost control matters here.
What to do next
The retrieval vs generation chapter covers the conceptual two-layer model in more depth. The faithfulness chapter covers atomic-fact decomposition and why "no hallucination" is the wrong frame. For dataset versioning, see the versioning chapter before you ship.