Positioning
Pydantic Evals is a Python library for evaluating non-deterministic functions. It is developed alongside Pydantic AI but uses it for only a small internal subset, and it is meant to work with any stochastic function, including code built on other agent frameworks. The README's stated priorities are type safety and ordinary Python over domain-specific constructs 1. The core vocabulary is three nouns: a Case (name, inputs, optional expected output and metadata), a Dataset (a named set of cases plus evaluators), and an Evaluator (a class with an evaluate method that receives the case, output and trace) 2.
The distinguishing choice is that every case runs under OpenTelemetry. The library records a trace per case and sends it to any compatible backend, and evaluators can read the span tree to grade the path the code took, not only the final string 1. As of October 2026 the docs also describe agent-trajectory evaluators and attaching evaluators to sampled production traffic 2. Datasets can also be defined in YAML or JSON and loaded from Python 2.
Figure: Pydantic Evals run path: a Dataset of Cases calls your task function once per case, and each call is traced as an OpenTelemetry span tree. Evaluators score the output and can also inspect the span tree, for example to check which tools were called. The EvaluationReport prints locally, and the same traces can be exported to Logfire.
Strengths
- Evals are ordinary Python. A task is a function, an evaluator is a dataclass, a suite is a script you can run under pytest or in CI. There is no separate runner, config language or server to stand up 1.
- Span-aware evaluators. Because each case is traced, a check like "did the agent call the refund tool exactly once" is an evaluator over the span tree rather than a log-scraping job. The 2.55.0 release fixed evaluators that run on errors so they get
span_tree and metrics when the task raises 3. That matters: failing cases are the ones you most want traces for.
- Useful built-ins without a key. The installed package exports
Equals, EqualsExpected, Contains, IsInstance, MaxDuration, LLMJudge, GEval, HasMatchingSpan, tool-call and trajectory evaluators, and report-level ones such as PrecisionRecallEvaluator and ROCAUCEvaluator (checked against 2.57.0). Only the judge-style ones need a model.
- No lock-in to a viewer. The report prints in the terminal. Logfire adds views for inputs, outputs, token usage and durations, but any OpenTelemetry backend receives the same traces 1.
Limitations
- Python-only, and it is a library, not a platform. There is no built-in annotation queue, shared dataset store or review UI. If product managers need to label traces, you are buying Logfire or another tool next to it.
- Release churn is high. The package went 2.55.0 (October 9, 2026), 2.56.0 (October 13) and 2.57.0 (October 16) in eight days 3 4 5. 2.55.0 raised the floor to Python 3.11 in every package, so a 3.10 environment resolves to 2.54.0 or earlier 3. 2.57.0 added
dataset_name to EvaluationReport 5, and in the version we ran Dataset refuses to construct without a name argument, which older snippets on blogs omit. Pin the version.
- Pip metadata ties it to the Pydantic AI family. The wheel requires
pydantic-ai-slim and logfire-api even if you never use an agent 6. The docs say it does not depend on the full pydantic-ai package 2. Expect a larger dependency tree than a bare eval script suggests.
- Few worked metric recipes. The docs call evals an emerging art and science and describe the framework as flexible rather than opinionated 2. Judge prompts, rubrics, and agreement checks against human labels are yours to write and validate.
Best fit
Python teams that already run pytest and OpenTelemetry, and want eval cases to live in the repo next to the code. Strong for agent apps where the tool-call path matters, and for teams already on Pydantic AI or Logfire. Weak fit for non-Python stacks, or teams that want a hosted annotation workflow out of the box.
Getting started
from dataclasses import dataclass
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext, EqualsExpected, IsInstance
def classify(text: str) -> str:
return "refund" if "refund" in text.lower() else "other"
@dataclass
class ShortOutput(Evaluator[str, str]):
max_len: int = 10
def evaluate(self, ctx: EvaluatorContext[str, str]) -> bool:
return len(ctx.output) <= self.max_len
dataset = Dataset(
name="refund-router",
cases=[
Case(name="refund", inputs="I want a refund", expected_output="refund"),
Case(name="shipping", inputs="Where is my parcel?", expected_output="other"),
Case(name="miss", inputs="Money back please", expected_output="refund"),
],
evaluators=[EqualsExpected(), IsInstance(type_name="str"), ShortOutput()],
)
def task(text: str) -> str:
return classify(text)
report = dataset.evaluate_sync(task)
report.print(include_input=True, include_output=True)
Run with pip install pydantic-evals on Python 3.11 or newer. No API key is needed because the task is deterministic. The report in our run (pydantic-evals 2.57.0, October 17, 2026) showed three cases: refund and shipping passing all three assertions, and miss failing EqualsExpected because the keyword rule returned other for "Money back please", for an average of 88.9% of assertions passing. That is the point of the tool: the failing case is a row you can read, not a number. To see traces in Logfire, install pydantic-evals[logfire] and configure Logfire before calling evaluate_sync.
Pricing notes
Pydantic Evals is MIT-licensed 7 and free. Model calls for LLMJudge or your own task are paid to the provider. Logfire is optional and billed separately. As of October 2026, the Personal plan is free with 10M spans, logs or metrics per month, no overage (ingestion pauses at the cap), and 30-day retention. Team is 2 per additional million, and Growth is $249/month with up to 90 days retention 8. A suite of 500 cases with a few spans each uses a small fraction of the free allowance; a production-sampled online eval is where the volume goes.
Alternatives
Consider Promptfoo for config-driven CI regression and red-teaming, or DeepEval for a pytest-style framework with more prebuilt metrics. Consider Inspect AI for sandboxed and frontier-style agent evals. If you want trace storage, annotation and datasets in one hosted product, look at Langfuse or Arize Phoenix and run Pydantic Evals as the in-repo harness.
Citations and last verified
Verified 2026-10-17 against the Pydantic Evals docs and README, PyPI metadata for pydantic-evals 2.57.0, the GitHub release pages for v2.55.0, v2.56.0 and v2.57.0, the repository LICENSE file, the Logfire pricing page, and a local run of the snippet above in a fresh virtualenv.