Positioning
LangSmith is the eval and observability layer attached to the LangChain runtime. The product covers traces, prompt management, datasets, online and offline evaluators (human, code, LLM-judge, pairwise), thread-level online evaluators for multi-turn conversations 1, and an issue-detection agent called Engine. It runs as hosted SaaS, with self-hosted and hybrid options on the Enterprise plan 2. The strongest fit is teams whose application is already built with LangChain or LangGraph, where the SDK emits trace trees without additional instrumentation 1.
Two additions since spring 2026 matter for eval work. LangSmith Engine, launched in public beta in May 2026, watches production traces, clusters failures into named issues, diagnoses root causes against your code, and proposes fixes, online evaluators, and dataset examples for the offline suite 3. It bills in LSUs at $1 each; the docs put initial setup at roughly 45 to 60 LSUs and recurring scans at 15 to 22.5, with monthly spend limits per workspace and per project 4. Tuned Evaluators, announced August 18, 2026, are hosted judges trained for one task. The first is Perceived Error, which flags threads where the user appears to have seen the agent make a mistake. It is in beta, limited to Plus and Cloud Enterprise in the US, and only runs on threads with at least two traces 5 6. LangChain's claim that it beat every frontier model in its own benchmark at 82% lower cost is vendor-published; calibrate it against your own labeled threads before trusting it.
Strengths
- First-class LangChain integration. Trace graphs, tool calls, and prompt templates show up in the UI without extra instrumentation if your app uses LangChain or LangGraph. SDKs exist for Python, TypeScript, Go, and Java 7.
- Mature eval primitives. Built-in evaluators for LLM-judge scoring, pairwise comparison, code evaluators, and a dataset interface tied to traces. The judge-alignment workflow is documented end-to-end including the human-feedback round-trip 8.
- Hosted, hybrid, and self-hosted options. The Enterprise plan covers self-hosted and hybrid deployments, for teams that cannot send traces to a vendor cloud 2.
Limitations
- Best returns inside LangChain. Teams on Vercel AI SDK, custom orchestration, or non-LangChain agent frameworks get less native value than they would from a framework-agnostic platform like Braintrust.
- Hosted-only for most plans. The pricing page lists self-hosting as unavailable on Developer and Plus, so it is enterprise-priced. Developer is capped at one seat; smaller teams pay $39 per seat per month for the cloud product on Plus 2.
- Vendor self-interest in benchmarks. Performance and judge-quality claims published by LangChain are vendor-published; treat them as positioning, not independent measurement.
Best fit
Offline regression suites built around LangChain runs, production observability for LangChain or LangGraph agents, and dataset versioning where the dataset and trace graphs share a single source of truth. If your codebase already imports langchain or langgraph, LangSmith is the path of least resistance.
Getting started
from langsmith import Client
client = Client()
# A dataset of inputs and reference outputs.
dataset = client.create_dataset(
dataset_name="dietary-restrictions-v1",
description="Tickets where the user states a dietary restriction.",
)
client.create_examples(
dataset_id=dataset.id,
examples=[
{
"inputs": {"question": "Is this vegan?"},
"outputs": {"answer": "Yes, the lentil bowl is vegan."},
}
],
)
# Your application under test.
def app(inputs: dict) -> dict:
return {"answer": my_agent(inputs["question"])}
# An LLM-as-judge evaluator.
def judge(outputs: dict, reference_outputs: dict) -> dict:
score = run_judge_prompt(outputs, reference_outputs)
return {"key": "answer_correctness", "score": score}
client.evaluate(app, data=dataset.name, evaluators=[judge])
The hosted UI then shows the per-example trace, the judge score, and a diff against any previous run on the same dataset 9. Older examples import evaluate from langsmith.evaluation, which the SDK marks deprecated since 0.5.0 in favor of client.evaluate(), and pass inputs= and outputs= lists to create_examples, which the SDK now documents as legacy arguments since examples= arrived in 0.3.11.
Pricing notes
The pricing page (fetched as of 2026-10-01) lists Developer at 39 per seat per month with up to 10k base traces and one free small Serverless deployment; and Enterprise at custom pricing with self-hosted and hybrid options 2. Usage past the allowance is pay-as-you-go, metered in LangChain Standard Units (LSU) at $1.00 each. Base traces keep 14 days of retention and extended traces keep 180 days, for an upgrade fee. The text of the pricing page does not state a per-trace LSU rate, so estimate steady-state trace volume with the vendor's cost calculator before committing to a contract.
Alternatives
Consider Braintrust if you want the same eval surface without LangChain-specific gravity. Consider Langfuse if you need a genuinely self-hostable open-source platform. Consider Arize Phoenix if your stack is OpenTelemetry-native and you want OSS-first. For pure CI regression with no observability, DeepEval or Promptfoo cover the offline case more cheaply.
Citations and last verified
Verified 2026-10-01 against the LangSmith pricing page, product page, changelog, Engine and Tuned Evaluators docs, the LangSmith documentation, the LangChain evaluation-concepts page, and the langsmith Python SDK source.