Positioning
Braintrust is a hosted observability and eval platform whose central claim is no framework lock-in: traces, datasets, and experiments work the same whether your app is built on LangChain, the Vercel AI SDK, the OpenAI SDK directly, or a custom orchestration layer. The proprietary Brainstore search engine became the default trace store in March 2025 1, and Loop, the built-in assistant that creates and edits prompts, scorers, and datasets, is available to all users per a November 2025 launch post 2.
Starter and Pro are SaaS only, and the product is not open source. The Enterprise plan adds two deployment modes: self-hosting, where you run the data plane (API, Postgres, Redis, object storage, Brainstore) in your own cloud and Braintrust runs the control plane, and BYOC, where Braintrust operates everything inside your cloud account 3 4.
In September 2026 Braintrust put three connected features into public preview: Patterns, a Loop automation that scans production traces for recurring issues, Debugger, which analyzes a single complex trace to find the likely failure mode, and a broader Loop that investigates across projects, runs scheduled automations, and asks for approval before changing evaluators or datasets. The announcement says pricing details will follow before general availability, so do not budget for these yet 5.
Strengths
- Framework-agnostic SDK. A small Python or TypeScript SDK wraps your calls and emits traces; no rewrite of agent code is required to switch frameworks.
- Strong dataset and experiment tooling. Side-by-side experiments, pairwise diff views, and a datasets API with versioning. Loop can bootstrap scorers from production traces and build datasets from problematic logs 2.
- Public customer roster. Coursera, Notion, Vercel, Dropbox, Replit, and Cloudflare are among the customers named on Braintrust's customers page, which is a useful signal that the platform handles non-trivial production traffic. Dropbox has a case study on its 10,000-plus-test eval suite for AI search. Notion has written publicly about its LLM-as-judge eval program, though that post does not name Braintrust 6.
Limitations
- Self-hosting is Enterprise-only. Starter and Pro are SaaS. Teams with strict data-residency rules can get a customer-run data plane or BYOC, but only on the Enterprise plan; the alternative is a self-hostable open-source platform 4.
- Proprietary stack. Brainstore is a closed search engine; you cannot inspect or replicate the indexing strategy.
- Self-interested benchmarks. Braintrust publishes performance comparisons against named competitors. Treat those as vendor positioning, not independent measurement.
Best fit
Hosted offline regression suites, dataset versioning, and side-by-side experimentation for teams that want a single console for traces and evals without committing to a single agent framework. Strongest fit when time-to-value is the constraint and self-hosting is not a hard requirement.
Getting started
import braintrust
from braintrust import Eval
# Wrap any LLM client; traces stream to the Braintrust project.
client = braintrust.wrap_openai(openai.Client())
def task(input: dict) -> str:
return client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": input["question"]}],
).choices[0].message.content
def scorer(input: dict, output: str, expected: str) -> float:
return 1.0 if expected.lower() in output.lower() else 0.0
Eval(
"dietary-restrictions-v1",
data=lambda: [
{"input": {"question": "Is this vegan?"}, "expected": "vegan"},
],
task=task,
scores=[scorer],
)
The Braintrust UI then renders the experiment with the per-example trace and the score. Subsequent runs against the same dataset are diffed automatically. The current quickstart runs evals from the command line with bt eval <file> and calls braintrust.auto_instrument() instead of wrapping the client by hand 7; wrap_openai is still exported by the Python SDK.
Pricing notes
The pricing page (fetched as of 2026-10-01) lists Starter at 10 of model credits, then 2.50 per 1,000 scores past those limits. Pro is 100 of model credits, custom charts, environments, and RBAC, then 1.50 per 1,000 scores, with extended retention at $0.50 per GB per month 8. Users, projects, and datasets are unlimited on both. Enterprise is custom and is the tier with SAML or OIDC SSO, audit logging, BYOC, and self-hosting 4. Like other hosted platforms, costs scale nonlinearly with trace volume, so estimate a steady-state monthly trace count before signing a multi-year contract; teams that build aggressive sampling into their pipeline pay much less than those that send every production trace.
Alternatives
Consider LangSmith if your stack is LangChain-heavy. Consider Langfuse when you need open-source self-hosting without an enterprise contract. Consider Arize Phoenix for an OpenTelemetry-native OSS hub. For pure CI regression with no observability, DeepEval and Promptfoo are cheaper.
Citations and last verified
Verified 2026-10-01 against the Braintrust pricing page, plans-and-limits and self-hosting docs, customers page, changelog, Loop and active-observability blog posts, and the evaluation quickstart.