Positioning
Opik is the open-source LLM eval and observability project from Comet, the experiment-tracking company. The codebase is Apache 2.0, which makes it the most permissive license among the major OSS eval hubs, and the project ships frequently: v2.2.87 was tagged October 1, 2026, and the repo shows 22.3k stars (as of 2026-10-01) 1.
The product covers tracing, prompt management, datasets, 30+ LLM-judge metrics (hallucination, contextual relevance, moderation, conversational quality, and more), a prompt optimizer with six search algorithms, and "Ollie," a conversational assistant built into the Opik UI. Ollie investigates traces and finds root causes; once you run opik connect it can also read your local source, propose code edits for your approval, and rerun the agent against your test suites 2.
The Opik evaluation docs now lead with Test Suites. You write expected behavior as natural-language assertions, an LLM judge checks each one against the agent's output and reports pass or fail, and an execution policy sets run frequency and pass thresholds. Assertions apply to the whole suite or to individual items 3. Datasets plus numeric metrics remain the path for scoring quality at scale.
Strengths
- Apache 2.0 license. No commercial-use restriction, no enterprise-only modules in the OSS core. Resellable, embeddable, and self-hostable without legal review. Permissive licensing is one of the differentiators flagged in surveys of popular open-source AI tools 4.
- Wide metric library. Out-of-the-box LLM-judge metrics cover the most common cases (hallucination, faithfulness, relevance, conversational quality, moderation, custom rubric). The prompt optimizer adds an experimentation loop on top.
- Public customer roster. AssemblyAI, NatWest, Stellantis, Uber, Zencoder, Netflix, Autodesk, Etsy, Stability AI, and Mobileye are listed as customers on the product page (as of 2026-10-01). The roster signals real production traffic and an active community.
Limitations
- Younger than Phoenix and Langfuse. The release cadence is high, and GitHub stars no longer separate the projects (Opik 22.3k, Phoenix 11.7k, as of 2026-10-01). Some integrations that exist for Langfuse and Phoenix are not yet first class, so check that the one you need is.
- Comet brand confusion. Opik is a separate product from Comet's experiment-tracking core. Teams onboarding the OSS hub sometimes confuse the two SKUs.
- Vendor-published benchmarks. Performance comparisons from Comet are vendor-published; treat them as positioning.
Best fit
OSS observability hubs where Apache 2.0 is preferred over ELv2 or MIT-with-EE, RAG evaluation with the Opik metric library, and hybrid setups where the OSS hub runs alongside Promptfoo or DeepEval in CI.
Getting started
import opik
from opik.evaluation.metrics import Hallucination, AnswerRelevance
from opik.evaluation import evaluate
@opik.track # auto-traces the function and its LLM calls
def app(question: str, context: list[str]) -> str:
return run_rag(question, context)
def task(item: dict) -> dict:
# evaluate() needs a dict whose keys match the metric parameters
return {
"input": item["question"],
"output": app(item["question"], item["context"]),
"context": item["context"],
}
dataset = opik.Opik().get_or_create_dataset("rag-dev")
dataset.insert([
{"question": "Is the dish vegan?", "context": ["Lentil bowl: no dairy."], "expected": "yes"},
])
evaluate(
experiment_name="rag-v1",
dataset=dataset,
task=task,
scoring_metrics=[Hallucination(), AnswerRelevance()],
)
The trace, the metric scores, and the per-item judge reasoning render in the Opik UI. Subsequent runs against the same dataset diff automatically. The task must return a dict keyed by the names the metrics expect (input, output, context), not a bare string 5. This snippet was corrected against the current docs and was not run here.
Three changelog items matter for CI and cost control 6. opik.evaluate_resume() replays only the missing trials of an interrupted experiment (June 2026). Passing error_tolerance=ErrorTolerance.ALL_SCORING_ERRORS to evaluate() records pre-scoring failures as failed scores instead of aborting the run (August 2026). And online LLM-judge rules take a max cost per evaluation in USD; a run that hits the cap returns a best-effort verdict and is tagged budget_exceeded (July 2026).
Pricing notes
Opik OSS is free under Apache 2.0 with unlimited spans, retention, and seats when you self-host. Comet Cloud lists a Free tier at 19 per month (100k spans per month, 60-day retention, up to 50 members), and custom Enterprise with flexible deployments (fetched as of 2026-10-01) 7. Teams that already license Comet for experiment tracking often find Opik bundled at favorable rates; teams that buy Opik standalone should price-check against Langfuse Pro and the Arize AX tiers before committing. The Apache 2.0 license is the strong tiebreaker when legal review is on the critical path.
Alternatives
Consider Langfuse if MIT plus EE modules are preferred to Apache 2.0. Consider Arize Phoenix for OpenTelemetry-native instrumentation. Consider LangSmith or Braintrust for hosted-only alternatives. For pure CI regression, DeepEval or Promptfoo.
Citations and last verified
Verified 2026-10-01 against the Opik GitHub repository and PyPI listing, the Opik product and pricing pages, and the Opik docs (evaluation concepts, evaluate function, Ollie, changelog).