Positioning
DeepEval is an open-source LLM testing framework styled after pytest: tests live alongside code, run on every PR, and read like unit tests. The library covers G-Eval (rubric-driven LLM judge), hallucination, RAG faithfulness and contextual relevance, agentic tool-call evaluation, and multi-turn conversational metrics. Red teaming lives in DeepTeam, a separate package built on DeepEval. Confident AI is the hosted layer that adds dashboards, dataset versioning, and team workflows on top of the OSS library 1.
The G-Eval rubric pattern comes from the original G-Eval paper 2; DeepEval's implementation is one of the most-used in production. Confident AI's homepage carries a "Trusted by 500+ leading AI companies" logo grid that includes Panasonic, Toshiba, Samsung, and Epic Games; treat it as vendor positioning, not an audited customer list (as of October 2026) 3. DeepEval 4.0 shipped in May 2026 and the current release is 4.2.8 (October 2, 2026) 4.
Strengths
- Pytest ergonomics. Tests run with
deepeval test run, integrate with GitHub Actions, and fail the build on regressions. Developers who already write pytest do not learn a new harness.
- Wide metric library. G-Eval, hallucination, summarization, faithfulness, contextual relevance, answer relevance, bias, toxicity, tool correctness, task completion, and multi-turn conversational quality, all callable from the same API. The 4.x series added deterministic agent checks that need no judge call (
AgentLoopDetectionMetric, ToolPermissionMetric), a community CitationFaithfulnessMetric, and a flaky flag so a known-unstable metric's failures no longer fail the whole run 5.
- Active red-team suite. DeepTeam is a separate Apache-2.0 package built on DeepEval (
pip install deepteam). It ships framework presets for the OWASP Top 10 for LLMs 2025, the OWASP Top 10 for Agentic Applications 2026, NIST AI RMF, MITRE ATLAS, BeaverTails, and Aegis, which is a useful starting point for adversarial testing without rolling your own attack library 6.
- Opt-in non-generative judge. Since 4.2.6 (September 2026),
deepeval set-eval-mode picks who judges the built-in metrics, and a single metric can override it with an eval_mode argument. The default llm mode is unchanged. deepeval set-eval-mode hybrid keeps the LLM for extraction and the written reason and hands each verdict to Jev, TypeSafe AI's decision model, which returns probabilities instead of generated text; deepeval set-eval-mode system_one lets Jev run the whole metric with no LLM call; deepeval set-eval-mode llm switches back 7. G-Eval stays on the LLM in every mode, and Jev needs a separate TypeSafe API key. The cost and latency figures ($0.042 per million input tokens, roughly 100 ms per query) are vendor-published, the benchmark DeepEval promised against a fully LLM-driven eval workflow was not out as of October 2026, and the announcement says a system_one score is not interchangeable with the LLM-derived one, so rebaseline before comparing runs 8. Decision models as judges covers when that trade is worth making.
Limitations
- Hosted upsell. The OSS library is genuinely useful, but the most polished workflows (dataset versioning, team dashboards) live in Confident AI. Teams need to decide where to draw the line. Run history no longer forces the decision: since September 2026,
deepeval set-local-store sqlite writes every test run to a local SQLite database you can query with SQL, instead of one JSON file per run 9.
- LLM-judge bias still applies. G-Eval inherits the standard judge biases (position, verbosity, self-preference) flagged in the LLM-judge literature. Calibration against humans remains the user's responsibility.
- Score direction changed in 4.2.0.
BiasMetric, HallucinationMetric, MisuseMetric, and ToxicityMetric moved to the higher-is-better convention in August 2026. Thresholds written for the old direction now mean the opposite, so audit them when you upgrade 5.
- Vendor-published benchmarks. Performance and metric-quality claims from Confident AI are vendor-published.
Best fit
Offline regression suites in CI, RAG eval with the built-in faithfulness and contextual-relevance metrics, agentic tool-call evaluation, and red-team programs that want a structured starting point with the OWASP categories.
Getting started
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase
def test_rag_answer():
case = LLMTestCase(
input="Is the lentil bowl vegan?",
actual_output=run_rag("Is the lentil bowl vegan?"),
retrieval_context=["Lentil bowl ingredients: lentils, oil, salt."],
expected_output="Yes.",
)
assert_test(case, [
AnswerRelevancyMetric(threshold=0.7),
FaithfulnessMetric(threshold=0.8),
])
Run with deepeval test run test_app.py. The CLI prints per-test scores and a summary; results push to Confident AI if the project is configured.
Pricing notes
DeepEval is free under Apache 2.0. Confident AI's pricing page lists four plans as of October 2026: Free (200 per month; unlimited seats, 5 projects, online evals, annotation queues, alerting), Team (1 per GB-month 10. A cap of 5 test runs per week means per-PR CI that reports to the platform outgrows the Free plan quickly. The OSS library is genuinely standalone, so teams that want a CI regression suite without the hosted dashboard pay nothing for the framework itself; LLM-judge calls are billed per token to the underlying provider. Teams that need dataset versioning, team workflows, or compliance-grade reporting are the audience for the hosted SKU.
Alternatives
Consider Promptfoo for a CLI-first OSS regression suite. Consider Ragas for RAG-specific metrics. Consider Inspect AI for sandboxed agentic eval. For hosted dashboards without DeepEval, Braintrust, Langfuse, or LangSmith.
Citations and last verified
Verified 2026-10-02 against the DeepEval GitHub repository, eval-modes docs and 2026 changelog, the 4.2.x packages on PyPI, the Jev and SQLite local-storage announcements, the DeepTeam README, and the Confident AI homepage and pricing page.