Positioning
Ragas began as the reference implementation of the RAGAS paper 1 and has grown into the de facto Python library for RAG evaluation. The metric set covers faithfulness, answer relevancy, context precision, context recall, context entity recall, and answer correctness, plus a synthetic test-set generator that builds query-context-answer triples from a corpus. The library is provider-agnostic (any LLM and any embedding model) 2.
Version 0.4 (December 2025) moved the library from one-off metric runs to an experiment-centric API. Metrics now live in ragas.metrics.collections, take keyword arguments, and return a MetricResult with a .value and an optional .reason. evaluate() still runs but is deprecated in favor of an @experiment decorator. Most 0.3 code keeps running with deprecation warnings. The removals are instructor_llm_factory(), which llm_factory() replaces, and the AspectCritic, SimpleCriteria, and AnswerSimilarity metrics, which the collections API dropped 3. The 0.4.3 source names v1.0 as the release where the old ragas.metrics imports go away 4.
Maintenance has slowed. As of October 2026, 0.4.3 (January 13, 2026) is the latest release, the repository (now under the vibrantlabsai GitHub organization) has had no commits to main since February 24, 2026, and two community PRs that fix the broken import described under Limitations have sat unmerged since June and August 4 5. The metric definitions are stable, so this matters less for the math than for dependencies: pin versions and expect to carry your own patches.
The fit is teams who need RAG-specific metrics in CI or a notebook and want to pair them with a tracing hub. Ragas does not provide a UI or production observability; that is intentional.
Strengths
- Definitive metric implementations. Faithfulness is implemented as atomic-claim decomposition plus entailment against the retrieved context, which matches the published methodology 6. Context precision and recall are the standard retrieval-quality measures.
- Provider-agnostic, with structured results.
llm_factory wraps an OpenAI, Anthropic, Google, Azure, Bedrock, or other client, and every score comes back with an optional reason 3. Experiment results export with to_pandas() 4. Fits notebook workflows and CI pipelines.
- Synthetic test-set generation.
TestsetGenerator builds a knowledge graph over your documents and synthesizes single-hop and multi-hop queries, each with reference contexts and a reference answer, useful for bootstrapping evals before human labels exist 7.
Limitations
- Fresh installs fail to import. As of October 2026,
pip install ragas pulls ragas 0.4.3 with langchain-community 0.4.2 (May 22, 2026), which removed langchain_community.chat_models.vertexai, and import ragas raises ModuleNotFoundError. The issue is open with no fixed release 5. Pinning langchain-community<0.4.2 restored the import in a clean virtualenv in October 2026.
evaluate() and the new metrics do not mix. In 0.4.3, evaluate() accepts only the legacy metric classes. Passing a ragas.metrics.collections metric raises TypeError, even though the migration guide's "before" example does exactly that 4 3. Pick one path: legacy metrics through evaluate(), or collections metrics through @experiment or direct ascore() calls.- Experiments drop failed rows.
arun catches an exception on any row, prints a warning, and leaves that row out of the results. A judge timeout on 3% of rows shrinks the denominator instead of failing the run, so check the result row count against the input before you read a mean 4.
- Reference-free metrics inherit judge bias. Faithfulness and answer relevancy depend on an LLM judge. The standard biases (position, verbosity, self-preference) apply; calibration against humans is the user's job.
- No UI or trace store. Ragas is a metric library, not a platform. Pair with Langfuse, Phoenix, Braintrust, or another hub to visualize trends and persist runs.
- Compute cost on large eval sets. Each metric is one or more LLM calls per row. In the 0.4.3 code, the four metrics in the snippet below make 6 + k judge calls per row for k retrieved chunks: two for faithfulness, three for answer relevancy (
strictness=3, plus two embedding calls), one per chunk for context precision, and one for context recall. A 10,000-row set with five chunks per row is 110,000 judge calls before retries 4. Cache aggressively.
- Telemetry is on by default. Ragas sends anonymized usage events to its own endpoint unless
RAGAS_DO_NOT_TRACK=true is set. Set it in CI and anywhere with data-egress rules 4.
Best fit
RAG evaluation in CI, RAG eval programs in a notebook, synthetic test-set construction from a corpus, and hybrid setups where Ragas runs as the metric source and Langfuse or Phoenix is the trace and dashboard hub.
Getting started
# pip install "ragas==0.4.3" "langchain-community<0.4.2" (see Limitations)
import asyncio
from openai import AsyncOpenAI
from pydantic import BaseModel
from ragas import Dataset, experiment
from ragas.embeddings import OpenAIEmbeddings
from ragas.llms import llm_factory
from ragas.metrics.collections import (
AnswerRelevancy,
ContextPrecision,
ContextRecall,
Faithfulness,
)
client = AsyncOpenAI()
judge = llm_factory("gpt-4o-mini", client=client)
embeddings = OpenAIEmbeddings(client=client, model="text-embedding-3-small")
faithfulness = Faithfulness(llm=judge)
answer_relevancy = AnswerRelevancy(llm=judge, embeddings=embeddings)
context_precision = ContextPrecision(llm=judge)
context_recall = ContextRecall(llm=judge)
dataset = Dataset(name="rag_smoke", backend="inmemory")
dataset.append({
"user_input": "Is the lentil bowl vegan?",
"response": run_rag("Is the lentil bowl vegan?"),
"retrieved_contexts": ["Lentil bowl ingredients: lentils, oil, salt."],
"reference": "Yes.",
})
class Scores(BaseModel):
user_input: str
faithfulness: float
answer_relevancy: float
context_precision: float
context_recall: float
@experiment(Scores)
async def score_row(row):
q, a = row["user_input"], row["response"]
ctx, ref = row["retrieved_contexts"], row["reference"]
f = await faithfulness.ascore(user_input=q, response=a, retrieved_contexts=ctx)
r = await answer_relevancy.ascore(user_input=q, response=a)
p = await context_precision.ascore(
user_input=q, reference=ref, retrieved_contexts=ctx
)
c = await context_recall.ascore(
user_input=q, retrieved_contexts=ctx, reference=ref
)
return Scores(
user_input=q,
faithfulness=f.value,
answer_relevancy=r.value,
context_precision=p.value,
context_recall=c.value,
)
results = asyncio.run(score_row.arun(dataset))
print(results.to_pandas())
This is the 0.4 shape: collections metrics scored inside an @experiment function, one result row per input row. to_pandas() gives per-row scores; the aggregate is a .mean() away, once you have checked that no rows were dropped. AnswerRelevancy needs an embeddings client as well as the judge, and both take an async client. The Ragas RAG getting-started guide still builds this loop on the deprecated evaluate() path with a LangChain wrapper; the v0.3 to v0.4 migration guide maps each piece to the new API 8 3. Existing evaluate() code still runs on 0.4.3, including datasets with the old question, answer, contexts, and ground_truth column names, but plan the move before v1.0.
Pricing notes
Ragas is free under the Apache 2.0 license. LLM-call costs (for faithfulness, relevancy, and similar reference-free metrics) are paid to the underlying provider. Teams can hold judge cost steady by capping the eval set size, picking a cheaper judge model once it is calibrated against humans, and caching judgments by content hash so prompt edits do not re-run unchanged rows. For faithfulness, FaithfulnesswithHHEM replaces the claim-verification LLM call with Vectara's HHEM-2.1-Open classifier, run locally on CPU or GPU 6. Claim extraction still uses the LLM, and as of 0.4.3 the class exists only on the legacy API, not in collections. It loads the model from Hugging Face with trust_remote_code=True, which some security reviews will flag 4.
Alternatives
Consider TruLens for feedback functions that compose, judge-calibration tooling, and a project that still ships releases every few weeks. Consider DeepEval for pytest-style ergonomics with overlapping RAG metrics. Consider LlamaIndex evals if your stack is already on LlamaIndex. For tracing and dashboards, pair with Arize Phoenix, Langfuse, or Braintrust.
Citations and last verified
Verified 2026-10-02 against the Ragas documentation and v0.3 to v0.4 migration guide, the GitHub repository, releases and issue tracker, and the 0.4.3 package installed in a clean virtualenv.