MLflow GenAI Evaluation

MLflow's mlflow.genai module, reviewed: evaluate(), 24 built-in scorers, make_judge(), and tracing in one Apache-2.0 platform, plus the heavy-server cost and the new Decision API route for judges.

Positioning

MLflow started as an experiment tracker for classical ML and is now also an LLM evaluation and tracing platform. The code is Apache-2.0 and the repository shows 28.4k stars (as of October 2026) ¹. Databricks created it and handed it to the Linux Foundation in June 2020, and Databricks still sells a managed version ². The GenAI side lives under mlflow.genai: evaluate() runs a dataset through your app, scorers grade the result, and the feedback is attached to the trace of each call ³.

The design choice that matters: evaluation and tracing share one data model. A scorer can read the output string, or the full trace of spans, so grading a tool-using agent does not need a second instrumentation layer. Teams that already log runs to MLflow get evals without adding a vendor.

Figure: MLflow GenAI evaluation flow: evaluate() takes a dataset of inputs and expectations and calls your predict_fn on each row, which is traced into the tracking store. Scorers (plain Python, built-in judges, or make_judge) grade the outputs and traces, calling a judge model when the scorer is LLM-based. Results land as a run with one metric per scorer, and per-row feedback sits on each trace.

Strengths

One system for traces and scores. Feedback is stored on the trace, so you can filter production traces by a failing scorer and click through to the spans. Most OSS eval libraries make you join two tools to get this.
Plain-Python scorers. @scorer turns any function over inputs, outputs, expectations or trace into a scorer, returning a bool, number, string or Feedback. Since 3.16.1 the decorator takes a timeout option ⁴; in 3.17.1 the default for @scorer scorers is 300 seconds, and 0 disables it.
A fixed set of built-in judges. mlflow.genai.scorers exports 24 built-in scorer classes in 3.17.1, among them Correctness, Guidelines, Safety, RetrievalGroundedness, ToolCallCorrectness, UserFrustration and a non-LLM RegexMatch ⁵. That is a working starter set, not an exhaustive metric catalog.
Judges you can swap models under. make_judge() takes a model URI, instructions with {{ inputs }}, {{ outputs }}, {{ expectations }}, {{ trace }} or {{ conversation }} placeholders, and a feedback_value_type such as Literal["yes", "no"] ⁵.
Decision-model judges. 3.17.0 shipped typed Boolean or categorical judging with TypeSafe AI's Jev across evaluation, gateway and tracing ⁴, and 3.17.1 added Decision API support to make_judge() ⁴. In the installed code, ModelAPI.DECISIONS selects OpenAI's Decision API or TypeSafe's System One API depending on the model provider, and a gateway:/ endpoint can also serve System One judges (TypeSafe or OpenRouter Jev) ⁵ ⁴. Background: Decision models as judges covers Jev ⁶, and OpenAI's Decisions API vs Jev covers the OpenAI side ⁷.

Limitations

It is a platform, not a library. The review UI needs a tracking server or a local SQLite file, and results live in that store rather than in your repo. For a CI check that just fails on regressions, Promptfoo or a pytest file is lighter.
Server upgrades are not free. 3.17.0 requires a coordinated schema upgrade for SQL-backed servers: stop all writers, back up, run mlflow db upgrade, and mixed-version rolling upgrades are not supported ⁴. Plan that before you pin a shared server to a new minor version.
Decision judges give up the rationale. The make_judge docstring says decision models return probabilities but no rationale, and do not support {{ trace }} tool calling ⁵. A Decision API judge cannot read your agent's spans, and a failed row comes with a probability and no explanation. Use them for cheap, high-volume checks on the final answer, and keep a chat-based judge for trace-level grading.
Fast minor releases. Three of the four releases between September 17 and October 13, 2026 touched evaluation ⁴. Pin the version, and re-score a fixed gold set after every bump, as you would for any judge change.
Your predict_fn runs once before the real run. evaluate() tests the function on the first row to validate tracing. If your app has side effects, that surprises people; set MLFLOW_GENAI_EVAL_SKIP_TRACE_VALIDATION=true to skip it ⁵.

Best fit

Teams that already track experiments or models in MLflow, Databricks shops, and groups that want traces, datasets, scorers and a review UI under one open-source roof they can host themselves. Weaker fit for a single-developer prompt regression suite, where setup cost dominates.

Getting started

import mlflow
from mlflow.entities import Feedback
from mlflow.genai import evaluate
from mlflow.genai.judges import make_judge
from mlflow.genai.scorers import scorer

mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("demo")


@mlflow.trace
def app(question: str) -> str:
    # Replace with your real app call.
    return {"What is 2+2?": "4", "Capital of France?": "Paris is the capital."}.get(question, "unknown")


@scorer(timeout=30)
def contains_expected(outputs, expectations) -> Feedback:
    ok = expectations["expected_response"].lower() in outputs.lower()
    return Feedback(value=ok, rationale="substring match")


@scorer
def short_enough(outputs) -> bool:
    return len(outputs) < 40


data = [
    {"inputs": {"question": "What is 2+2?"}, "expectations": {"expected_response": "4"}},
    {"inputs": {"question": "Capital of France?"}, "expectations": {"expected_response": "Paris"}},
]

results = evaluate(data=data, predict_fn=app, scorers=[contains_expected, short_enough])
print(results.metrics)  # {'contains_expected/mean': 1.0, 'short_enough/mean': 1.0}

I ran this against mlflow==3.17.1 in a fresh virtualenv with no API key: both scorers returned a mean of 1.0 and the run logged to SQLite. To add an LLM judge, create one with make_judge(name="polite", instructions="Is the reply in {{ outputs }} polite for the question in {{ inputs }}?", model="openai:/gpt-4.1-mini", feedback_value_type=Literal["yes", "no"]) and put it in scorers; it needs a provider key. Pass model_api=ModelAPI.DECISIONS (from mlflow.genai.judges) to route through a decision model. I constructed that judge without a key; I did not call it. Browse results with mlflow server --port 5000 ³.

Pricing notes

MLflow is free under Apache-2.0 and you pay your model provider for judge calls. Self-hosting costs a tracking server, a database and an artifact store. Databricks sells a managed MLflow inside its platform; I did not verify its pricing, so check your workspace rate.

Alternatives

Consider Promptfoo for CLI-first CI regression, Inspect AI for sandboxed agentic evals, and Langfuse or Arize Phoenix when you want a trace-first hub without the wider MLflow model-lifecycle surface. If you already run MLflow for training, adding mlflow.genai is the lowest-friction option.

Citations and last verified

Verified 2026-10-17 against the MLflow GitHub releases (3.16.1, 3.17.0, 3.17.1), the mlflow==3.17.1 package installed from PyPI (source and docstrings read, example executed), the MLflow GenAI evaluation docs, and the repository page.

MLflow, GitHub repository, https://github.com/mlflow/mlflow. ↩
Databricks, "MLflow joins the Linux Foundation," https://www.databricks.com/blog/2020/06/25/mlflow-joins-the-linux-foundation-to-become-the-open-standard-for-machine-learning-platforms.html. ↩
MLflow Docs, "Evaluating GenAI apps," https://mlflow.org/docs/latest/genai/eval-monitor/. ↩ ↩²
MLflow, "Releases," https://github.com/mlflow/mlflow/releases. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
MLflow, "mlflow 3.17.1 on PyPI," https://pypi.org/project/mlflow/. ↩ ↩² ↩³ ↩⁴ ↩⁵
TypeSafe AI, "Introducing System One Models and Jev," https://typesafe.ai/blog/introducing-system-one-models-and-jev. ↩
OpenAI, "Decisions guide (beta)," https://developers.openai.com/api/docs/guides/decisions. ↩