Positioning
The llama_index.core.evaluation module is LlamaIndex's first-party home for RAG evaluation. It ships LLM-judge evaluators for correctness, semantic similarity, faithfulness, context relevancy, answer relevancy, guideline adherence, and pairwise comparison, a BatchEvalRunner to run several of them concurrently, and a RetrieverEvaluator that scores retrieved node ids with hit rate, MRR, precision, recall, average precision, and NDCG. The evaluation guide lists integrations with UpTrain, Tonic Validate, DeepEval, Ragas, RAGChecker, and Cleanlab for teams that want a different metric library without leaving the LlamaIndex pipeline 1.
The dataset half of the module shrank in April 2026. PR #21294, first released in llama-index-core 0.14.21, deleted llama_index.core.llama_dataset (which held RagDatasetGenerator and LabelledRagDataset), the generate_question_context_pairs helper, and RetrieverEvaluator.aevaluate_dataset 2. The retrieval usage guide still shows the removed helper as of October 2026 3. What survives for synthetic questions is DatasetGenerator, which core marks deprecated in favor of the RagDatasetGenerator it no longer ships.
The fit is teams already on LlamaIndex for ingestion, chunking, and retrieval; the evaluators avoid a parallel toolchain. For teams not using LlamaIndex the integration story is less compelling than picking Ragas or DeepEval directly.
Strengths
- In-package integration. Retrievers, query engines, responses, and evaluators share the same abstractions.
evaluate_response takes a LlamaIndex Response and reads the source nodes for you, so there is no glue code between the pipeline under test and the judge.
- Retrieval metrics without a judge.
RetrieverEvaluator computes hit rate, MRR, and the other ranking metrics against expected node ids with no LLM call, which makes it the cheapest regression check in the module.
- Pluggable metric backends. Adapters mean you can pick the metric library that fits your taste without rewriting the pipeline. The Ragas paper's reference-free approach, judging faithfulness, answer relevance, and context relevance without gold answers, overlaps heavily with LlamaIndex's own faithfulness and relevancy evaluators 4. Bet on the actively maintained backends: UpTrain's last release is v0.7.1 from May 2024 5 and Tonic Validate's last tagged release is from November 2024 6.
Limitations
- Dataset tooling removed from core. If your harness imports
RagDatasetGenerator, LabelledRagDataset, or generate_question_context_pairs, it breaks on llama-index-core 0.14.21 or later. Pin below 0.14.21, keep using the deprecated DatasetGenerator, or move test-set generation to Ragas 2.
- Docs lag the code. The evaluation pages still link
LabelledRagDataset guides and show the removed helpers. Check imports against the installed version, not the docs 1 3.
- Tied to LlamaIndex abstractions. If your stack is built on a different orchestration framework (LangChain, raw OpenAI SDK, custom), the integration benefit disappears and you are better off with the metric library directly.
- Less polished UI than hosted hubs. LlamaIndex evals run from notebooks or scripts; there is no first-party dashboard. Pair with Phoenix, Langfuse, or Braintrust for visualization.
- Vendor-published benchmarks. Performance claims from the LlamaIndex project should be treated as vendor positioning.
Best fit
RAG evaluation for teams already building on LlamaIndex, retrieval regression checks against a labeled set of expected nodes, and pipelines that want to keep the eval logic in the same Python module as the retrieval and generation code. For synthetic test-set generation, look elsewhere.
Getting started
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.evaluation import (
DatasetGenerator,
FaithfulnessEvaluator,
RelevancyEvaluator,
)
# Index a corpus and build a query engine.
docs = SimpleDirectoryReader("data/").load_data()
index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine()
# Generate synthetic questions from the corpus. DatasetGenerator is marked
# deprecated, but RagDatasetGenerator was removed in llama-index-core 0.14.21.
generator = DatasetGenerator.from_documents(docs, num_questions_per_chunk=2)
questions = generator.generate_questions_from_nodes()
# Evaluate the pipeline.
faithfulness = FaithfulnessEvaluator()
relevancy = RelevancyEvaluator()
for q in questions:
response = query_engine.query(q)
f = faithfulness.evaluate_response(response=response)
r = relevancy.evaluate_response(query=q, response=response)
print(q, f.passing, r.passing, f.feedback)
Each call returns an EvaluationResult with passing, score, and feedback fields; collect them into a dataframe or push them to a hub for visualization. The snippet was checked against the llama-index-core 0.14.25 source; it was not run here. The judges default to Settings.llm, so set that explicitly before you trust the numbers.
Pricing notes
LlamaIndex is free under the MIT license. The hosted platform formerly called LlamaCloud is now sold as LlamaParse 7, with Free (10K credits a month), Starter (500 a month), and custom Enterprise plans (fetched as of 2026-10-02) 8. It covers document parsing, extraction, and indexing, not evaluation. The eval module runs locally with no service dependency, so the only direct cost is the per-token bill for the LLM judge used by the faithfulness, relevancy, and correctness evaluators.
Alternatives
Consider Ragas for RAG-specific metrics and test-set generation with no LlamaIndex coupling. Consider DeepEval for pytest-style ergonomics. Consider TruLens for feedback functions that compose. For tracing and observability, pair with Arize Phoenix or Langfuse.
Citations and last verified
Verified 2026-10-02 against the LlamaIndex evaluation guides, the llama-index-core 0.14.25 source and changelog, PR #21294, the UpTrain and Tonic Validate release pages, and the LlamaIndex pricing page.