The question every team asks once their JSONL passes a hundred lines is which vendor to put it in. The right answer for most teams is: keep the JSONL in git, sync it to one vendor for the UI, and resist the temptation to make the vendor the source of truth. The vendors are good at browsing, sharing, and annotation. They are not good at being a versioned content-hashed contract with your past self, and that is what an eval dataset is.
The source-of-truth principle
A dataset is a contract that lets you compare results across time. The substrate has to be diff-able, hash-able, reproducible from scratch, portable across vendors, and never silently mutated by a UI click. Git plus JSONL satisfies all of those. A SaaS dataset object that lives only in a vendor's database satisfies none. Hamel makes the same argument for prompts, which he prefers to keep in Git as versioned, reviewed artifacts deployed with the application [1]; the logic carries over to datasets.
The vendor UIs are still useful. They are excellent at annotation throughput, at letting non-engineers tag traces into the set, and at running scores in a hosted environment. Use them as the labeling surface and the downstream view; keep the JSONL canonical, and treat the vendor copy as a derived artifact you can rebuild from source.
How much vendor UI you need is a fair question. The FAQ (fetched as of 2026-10-01) recommends building your own annotation interface, which AI-assisted development tools can produce in hours, and keeping off-the-shelf tools for cases with dozens of distributed annotators and enterprise access controls. It also lists APIs that support custom annotation apps as a gap to check before you commit to a vendor, noting that bulk data export often needs hand-built libraries [1]. A custom viewer over the JSONL is one more reason the file, not the platform, stays canonical.
Figure: Eval dataset storage architecture: a JSONL file in git stays the source of truth, a CI sync script pushes it to Langfuse, Phoenix, Braintrust, or LangSmith on every content-hash change, and the vendor UI serves as a derived surface for browsing, annotation, and hosted scoring.
Capabilities below were checked against each vendor's docs on 2026-10-01. They move fast, so recheck before you commit.
Langfuse Datasets is the OSS-leaning option: an MIT-licensed core you can self-host [2], Python and TypeScript SDKs, dataset runs that pair each example with a model output and judge score, and a browse/annotate UI. Dataset items live in Postgres, the store Langfuse uses for transactional data [3]. Every add, update, delete, or archive of an item creates a new version tracked by timestamp, so you can fetch the dataset as it stood at a given moment, and you can attach JSON Schema validation to inputs and expected outputs [4]. ClickHouse acquired Langfuse in January 2026; the team says it stays open source and self-hostable with no planned licensing changes [5].
Arize Phoenix Datasets follows a similar pattern. Phoenix ships under the Elastic License 2.0, which Arize calls open source but which bars offering Phoenix to third parties as a hosted or managed service [6]. Every insert, update, and delete is versioned, and experiments (one run of system × judge × dataset version) pin to a specific dataset version [7]. Phoenix is heavier on the trace-ingestion side, so the natural flow is to capture traces in Phoenix, promote interesting traces into the dataset, and run experiments back against them [8].
Braintrust is a hosted, proprietary product oriented around the experiment-as-a-row workflow. Datasets are first-class and so are judges [9]. The UI is the most polished labeling surface of the four. Portability is the tradeoff: on Starter and Pro the data lives in their cloud and exporting means a script over the SDK, and running the data plane in your own cloud (self-hosted, or BYOC where Braintrust operates it for you) requires the Enterprise plan [10]. Datasets can be saved as snapshots to pin an experiment to a fixed version [11].
LangSmith Datasets is LangChain's hosted offering. Tightest integration with LangChain agents and a well-documented evaluation API [12] [13]. Dataset versioning is built in: every add, update, or delete creates a new version, versions can carry tags such as prod, and datasets download from the UI as CSV or JSONL [14]. Like Braintrust it is vendor-hosted by default, with self-hosted and hybrid deployments reserved for Enterprise [15].
Both commercial platforms now write dataset rows themselves. Braintrust's Loop, available to all users since its November 2025 launch, can generate a dataset from problematic production logs [16]. LangSmith Engine detects recurring failures in production traces and generates dataset examples from them for the offline suite [17]. Treat those rows as proposals: review them, and merge the accepted ones back into the JSONL through a pull request, or the vendor copy quietly becomes the source of truth.
Three axes for the decision
Open-source versus hosted. If you have a self-hosting team and prefer keeping data on your infrastructure, Langfuse and Phoenix. If you do not, Braintrust and LangSmith; both will run a data plane in your cloud, but only under an Enterprise contract.
Labeling persona. If non-engineers (PMs, domain experts, QA) need to label directly, the Braintrust or LangSmith UI pays for itself within a quarter. If labeling is engineer-driven and lives in notebooks, the Phoenix or Langfuse SDK flow is enough.
Lock-in tolerance. If you intend to switch judge platforms, observability vendors, or evaluation frameworks in the next two years (you should plan to), keep the JSONL canonical so you can repoint at a new vendor in an afternoon.
Do not spend long on feature matrices. Hamel's FAQ says the vendors' features are very similar, that in his client work the choice turns more on who offers the best support, and that any detailed comparison would be out of date within a week [1].
A sync script
# sync_dataset.py
import json, os
from pathlib import Path
from langfuse import Langfuse
DATASET, VERSION = "booking-flow-golden", "v3"
SOURCE = Path(f"evals/data/{VERSION}/golden.jsonl")
client = Langfuse(
public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
secret_key=os.environ["LANGFUSE_SECRET_KEY"],
base_url=os.environ.get("LANGFUSE_BASE_URL", "https://cloud.langfuse.com"),
)
def upsert():
try:
client.create_dataset(name=DATASET, metadata={"version": VERSION})
except Exception:
pass
for line in SOURCE.read_text().splitlines():
if not line.strip():
continue
r = json.loads(line)
client.create_dataset_item(
dataset_name=DATASET,
id=r["id"],
input=r["input"],
expected_output=r["expected_output"],
metadata={"version": VERSION, "labels": r.get("labels", {})},
)
The script runs as part of CI whenever the dataset hash changes. The vendor gets the same examples, addressed by the same stable ids, with the same labels. The UI becomes a window into the canonical store, not an alternative to it.
Two details from the SDK. create_dataset_item upserts when an item with the given id already exists, so rerunning the script is safe, and the id must be unique across the whole project rather than only within one dataset, so prefix ids with the dataset name if you sync more than one set [18]. The host argument is deprecated in favor of base_url, and the docs now set the URL with LANGFUSE_BASE_URL; the older LANGFUSE_HOST variable is still read as a fallback [19].
What stays out of the vendor
The content hash stays in git because diff-and-blame depends on it. MIGRATION notes between versions stay in git because they capture reasoning a future engineer needs. Result records stay in your warehouse or in the vendor's run store but reference the git hash by id; the warehouse copy is the audit trail.
A pragmatic recommendation
For a team starting today: JSONL in git, sync to Langfuse if you self-host and want OSS, sync to Braintrust if you want a polished labeling surface and accept a vendor-hosted data plane. Whatever the choice, the test of whether the architecture is right is: can you migrate to a different vendor in two weeks. If yes, the JSONL is genuinely canonical. If no, the vendor has quietly become the source of truth, and the next migration will be a multi-quarter project. Fix that before it hardens.
With the storage architecture settled, the harder question is what the rows have to measure once the system retrieves before it generates: RAG Evals.