Opik Review: The Open-Source LLM Evaluation and Tracing Platform, Tested, With the Defaults You Should Change

Opik is Comet's open-source platform for tracing, evaluating and monitoring LLM applications and agents, and its strongest argument is the license: the whole platform, server and UI included, is Apache-2.0 and can be self-hosted without a commercial agreement. That makes it a natural default when a customer's data cannot leave their network. We installed the SDK, read the server and SDK code, and ran its metrics offline. Two findings shape how we would deploy it. The SDK sends error reports and product analytics to Comet by default, and the self-hosted server sends daily usage counts, so a locked-down deployment needs four settings changed. And the evaluation metrics need calibrating before anyone trusts a dashboard: two string-similarity metrics scored a factually wrong answer 0.96 and a correct one 0.45. Opik is a good platform. It does not decide what "good" means for your application, and neither does any other tool.

What Opik does

Opik covers the loop from development to production:

• Tracing. Decorate a function with @track or use one of the integrations (the README lists dozens, including Google ADK, LangChain, CrewAI, DSPy, Anthropic, Gemini, Bedrock and an OpenTelemetry path for other languages) and every call becomes a trace with nested spans: model calls, tool calls, retrieval steps.
• Evaluation. Datasets, experiments that run your app over a dataset, and metrics: string heuristics (exact match, contains, regex, Levenshtein, BLEU, ROUGE, BERTScore and more) and LLM-judge metrics (hallucination, answer relevance, context precision and recall, factuality, moderation, usefulness, G-Eval, trajectory accuracy, LLM juries).
• Production monitoring. Dashboards for scores, trace counts and token usage, and online evaluation rules that score sampled production traces with judge metrics.
• Prompt management, a playground, guardrails, a pytest integration for CI, and an Agent Optimizer SDK with evolutionary, few-shot Bayesian, GEPA, meta-prompt and parameter optimizers.
• An MCP server so Claude Code, Cursor, Codex and similar agents can read traces and run evaluations from chat.

As of 21 October 2026 the repository has about 22,800 GitHub stars, the platform is at version 2.2.114 and the Python SDK we installed is 2.2.113. The README claims the system handles more than 40 million traces a day; that is Comet's figure.

Self-hosting: free license, real operations

The Docker Compose deployment starts MySQL, Redis, ClickHouse, ZooKeeper and MinIO, then Opik's Java backend, a Python backend, the frontend, and optionally guardrails backends, Jaeger and an OpenTelemetry collector. One script (./opik.sh) brings it up locally; a Helm chart covers Kubernetes.

That is a normal stack for an analytics product, and it is a real operational commitment. ClickHouse holds traces and grows with span size, not just trace count. Before the first deployment, decide retention, backups and who is paged when ingestion falls behind. "Free to self-host" means free of license fees. The bill moves to the person who runs ClickHouse.

For many customer engagements that is still the right trade. A regulated customer that cannot send prompts and outputs to a vendor's cloud has three realistic options among the popular tools: an Apache-2.0 platform it can run itself (Opik), an MIT-core platform with paid enterprise modules (Langfuse), or a closed product with an enterprise self-hosting plan. Opik's own comparison table puts it that way, and we agree with the licensing facts in it. The rest of the table is the vendor's framing.

The defaults that phone home

We read the SDK's configuration and the server's. None of this is hidden; all of it is on by default, and a security review will ask about it.

We checked the server's daily report in the source: it carries counts and the version, not trace content. That is a reasonable product choice and an easy one to turn off. The SDK's default target matters more. A developer who installs the SDK and runs a script without configuring it sends its requests to Comet's cloud endpoint, so they leave the network. Put the four settings in the base image or the shared .env on day one.

Metrics: what we measured

Opik ships many string-similarity metrics, and they are useful for the jobs they were designed for. We ran two of them offline against a one-line reference answer to see how they behave on a support-bot answer where meaning matters.

Metric definitions: Opik LevenshteinRatio and ROUGE implementation. The scores above are from the offline experiment described here, not an upstream benchmark.

The negated answer, which tells a customer they cannot get a refund, scored 0.96 on Levenshtein ratio and 0.94 on ROUGE-L. The answer with the wrong number scored 0.98 and 0.88. The correct paraphrase scored 0.45 and 0.20. A Contains("30") check caught the wrong number and missed the negation. None of this is a flaw in Opik's implementation; it is what string similarity measures. A metric that rewards looking like the reference will rank a confident wrong answer above a correct one in different words. Use these metrics for format, exact identifiers, extraction fields and regression detection on templated output, not for correctness of free text.

The LLM-judge metrics are where meaning gets checked, and two details in their code matter.

The hallucination judge also grades against general knowledge. With context supplied, the judge's instructions say the output must not add information beyond the context or contradict it, and also that it "should not contradict well-established facts or general knowledge". For public-domain questions that is sensible. For an enterprise assistant whose source of truth is the customer's own documents (a bank's fee schedule, an insurer's policy wording, a company's leave rules), the customer's facts can differ from what the judge model believes is standard, and faithful answers get marked as hallucinations. For grounded enterprise RAG, use a context-only faithfulness prompt.

G-Eval scores depend on the provider. When the judge model returns token log-probabilities, Opik computes the score as a probability-weighted average over the score tokens, which is the method from the G-Eval paper and gives smoother, more stable scores. When the provider does not return log-probabilities, it falls back to the integer the model wrote. The same metric on two providers can therefore produce differently shaped score distributions. Keep the judge model fixed within any comparison.

Two more defaults to know: the SDK's default judge model is openai/gpt-5-nano (set by default_llm), a small and cheap model, and judge prompts wrap inputs in tags with an instruction to treat them as data, which is a sensible defense against answers that try to instruct the judge.

Our rule for any judge, from any platform, is the one in our LLM-as-a-judge concept: label 100 to 200 examples by hand, measure the judge's agreement with those labels, and only then put its score on a dashboard. Pin the judge model and store the prompt version with every score. An eval score without a judge version is a number nobody can explain six months later.

What to use it for

Agent debugging. Nested traces with tool calls and retrieval spans are the fastest way to see why an agent took fifteen steps instead of four. This is where every team gets value in the first week. See agent observability and telemetry for what to record.

Regression gates in CI. Datasets plus experiments plus the pytest integration give you an eval regression suite: a fixed set of cases that must not get worse when the prompt, model or retrieval changes. Start with 50 cases that represent real failures, not 2,000 synthetic ones.

Production sampling. Online rules can score a sample of live traces with judge metrics. Score a few percent of traffic, alert on the trend, and route low scores to human review rather than paging on single traces.

Comparing models before an upgrade. Run the same dataset through the current and candidate model and compare per-case, not just in aggregate; regressions hide in averages.

What we would not use it for: as the source of truth for whether an answer is correct without human calibration, or as a security boundary. Guardrails help, but they belong in front of the model in your application, with their own tests.

Pros and cons

How we would set it up for a customer

1. Decide where traces live before writing code. If they cannot leave the network, self-host and set the four settings above in the base image.
2. Trace first, evaluate second. Instrument the agent with spans for every model call, tool call and retrieval, and spend a week reading traces before choosing metrics.
3. Build the dataset from real failures. Every bug report and every bad trace becomes a case.
4. Choose metrics per field. Exact or regex checks for structured fields; a calibrated, context-only judge for free-text faithfulness; human review for the rest.
5. Version everything that produces a score. Judge model, prompt, metric version, dataset version.
6. Set retention and redaction. Traces contain customer data. Decide what is masked at capture, how long spans are kept and who can read them.

For forward deployed engineers, this is one of the most common first weeks on an engagement: the customer has a prototype, no traces and a vague sense that quality is "mostly fine". Standing up tracing and a 50-case regression set turns that into numbers you can defend. It is also a standard interview topic. Practice explaining offline versus online evaluation, why a judge needs calibration and how you would gate a model upgrade, with our MLOps questions and the agentic software factory guide, which covers model upgrades as deploys.