Visibility in this ranking can be paid for. Payment gives a vendor more visibility, higher up on the review pages; it never adds a vendor, and it never changes a word of the review. The largest vendors in LLM observability and evaluation tools
cannot hold places 1 to 3. Places 1 to 3 go to challengers, vendors outside the best-known brands in a category,
wherever the category has them; the established names follow from place 4 in their ranked order. How it works: placement disclosure ·
editorial process.
The 12 tools, reviewed
#1 Langfuse
Open-source tracing, prompts and evals from a Berlin company · Germany · langfuse.com
Open source OpenTelemetry Self-hostable
Langfuse earns first place because one product covers the whole loop and the buyer keeps options open. Traces, versioned prompts, datasets, experiments, annotation queues and dashboards sit in the same interface, and a free cloud tier lets a team start without a card.
- Pricing: Open source + paid tiers. Hobby is free (50k units, 30 days, 2 users). Free plan: open source.
- Best for: Teams wanting open source plus managed cloud
- Strengths: Hierarchical tracing; Prompt management and versioning; LLM-as-a-judge evaluators.
Where it falls short
The unit model needs attention: Core and Pro include 100k units, and a verbose agent can burn through that quickly. Free retention is 30 days with two users, which is tight for a team.
Wrong for
A team that wants one vendor to carry the whole contract, with a gateway, guardrails and a managed security review included.
Visit Langfuse →
#2 Opik
Apache-2.0 tracing and evals, backend included · United States · comet.com
Apache-2.0 Agent tracing Self-hostable
Opik takes second place on openness and price. Comet's README states that the repository holds the backend, web application, tracing, datasets, experiments and online evaluation under Apache-2.0, so no feature is held back for a hosted edition.
- Pricing: Open source + paid tiers. Open source is free to run. Free plan: open source.
- Best for: Teams that must self-host the whole stack
- Strengths: Agent tracing and execution graphs; Test suites and assertions; Agent playground.
Where it falls short
Opik is a newer project than several rivals and its community is smaller than Langfuse's. The free and Pro cloud tiers cap spans, and larger limits and retention are negotiated.
Wrong for
A buyer who wants a managed vendor with a dedicated gateway, a large partner ecosystem and a long enterprise track record in this exact category.
Visit Opik →
#3 Pydantic Logfire
One OpenTelemetry timeline for agents, services and hosts · United States · pydantic.dev
OpenTelemetry Python Usage-priced
Logfire sits third because it puts LLM traces on the same timeline as ordinary services, so a slow tool call and the database query behind it appear in one view. It is built on OpenTelemetry and works with Pydantic AI, LangChain, the OpenAI SDK or anything instrumented.
- Pricing: Usage-based. Personal is free with 10M records a month, hard-capped; Team is $49 a month with 10M records included and $2 per extra million. Free plan: free plan.
- Best for: Python teams already on Pydantic AI
- Strengths: Agent run tracing; Service and host monitoring; OpenTelemetry SDK support.
Where it falls short
The pricing page shows hosted plans, and we did not see a self-hosting option for the Logfire backend, so data residency choices are limited to what the vendor offers.
Wrong for
A team that needs a prompt registry, annotation queues for non-engineers and a dataset workbench as the main interface, or one that must run the backend inside its own network.
Visit Pydantic Logfire →
#4 Arize Phoenix
Local-first LLM tracing and evals under ELv2 · United States · phoenix.arize.com
Source-available OpenInference Local-first
Phoenix ranks fourth for people who want to see traces before they talk to a sales team. The command uvx arize-phoenix serve starts the app locally, instrumentation follows OpenTelemetry with the OpenInference conventions, and the evaluation library covers relevance, toxicity and retrieval quality. The GitHub repository shows over 12,000 stars and Arize says its monthly downloads exceed three million.
- Pricing: Open source + paid tiers. Phoenix is free to self-host. Free plan: open source.
- Best for: Data scientists who want a laptop-first start
- Strengths: Tracing for agents and RAG; Evaluators for relevance and toxicity; Experiments and datasets.
Where it falls short
Phoenix is licensed under the Elastic License 2.0, which is source-available rather than open source in the OSI sense and bars offering it as a service.
Wrong for
A company whose legal team requires an OSI-approved license, or a product team that wants a polished annotation interface for reviewers who never open a notebook.
Visit Arize Phoenix →
#5 Datadog Agent Observability
LLM span tracing inside the Datadog platform · United States · datadoghq.com
APM vendor Per LLM span Platform add-on
Datadog is a large incumbent in application monitoring, so it cannot hold a top-three place, and it lands fifth because the fit is narrow but real. A team that already runs Datadog for services and infrastructure can see an agent run next to the database and the queue that served it, with one login and one contract.
- Pricing: Usage-based. From $160 a month billed annually ($200 month to month) for 100K LLM spans. Free plan: free trial.
- Best for: Companies already standardized on Datadog
- Strengths: LLM span tracing; Evaluations and experiments; Playground and datasets.
Where it falls short
There is no free plan, only a trial, and the entry price is $160 to $200 a month before any extra spans. Trace retention is 15 days.
Wrong for
A team without an existing Datadog contract, a solo builder or a startup that wants a free tier and an open-source option should start with Langfuse, Opik or Phoenix.
Visit Datadog Agent Observability →
#6 Braintrust
Evaluation-first platform with unlimited users · United States · braintrust.dev
Evals first Unlimited users Hosted
Braintrust places sixth because its centre is the evaluation workflow, and every plan, even the free Starter, allows unlimited users, projects, datasets, playgrounds and experiments. That suits a company where designers, product managers and domain experts must all work in the same place as engineers.
- Pricing: Usage-based. Starter is $0 with 1 GB processed and 14-day retention. Free plan: free plan.
- Best for: Product teams running many experiments
- Strengths: Tracing; Experiments and datasets; Playgrounds.
Where it falls short
The bill has four meters: platform fee, model credits, processed data and scores, which makes forecasting harder than a flat fee. Free retention is 14 days and Pro adds 30.
Wrong for
A team that needs to run the platform on its own servers from day one, or one that wants to pay a predictable flat fee, should price the open-source engines first.
Visit Braintrust →
#7 LangSmith
Tracing, evals and deployment from the LangChain team · United States · langchain.com
LangChain Per seat Agent deployment
LangSmith sits seventh. It is the obvious pick for teams already writing agents with LangChain and LangGraph, because tracing, datasets, evaluators, the prompt hub and agent deployment come from one vendor, and the LLM gateway and sandboxes are billed in the same unit. Plus allows unlimited extra seats at $39 each.
- Pricing: Per seat + usage. Developer is $0 for one seat with 5k base traces. Free plan: free plan.
- Best for: Teams building on LangChain and LangGraph
- Strengths: Tracing; Monitoring and insights; Online and offline evals.
Where it falls short
The pricing page mixes seats, base traces and LSUs, so a monthly bill needs the vendor's calculator. The free Developer plan allows one seat, so a team pays at the first collaborator. Self-hosting is an Enterprise feature.
Wrong for
A team without LangChain in its stack, or one that wants unlimited free collaborators and a self-hosted option without an enterprise contract, would do better with Langfuse or Opik.
Visit LangSmith →
#8 Orq.ai
Dutch AI gateway with built-in tracing and budgets · Netherlands · orq.ai
Dutch company AI Gateway OpenTelemetry
Orq.ai takes eighth place as the entry for teams that want observability to come with control. Requests pass through one OpenAI-compatible endpoint, each becomes a trace with cost, tokens, latency and payloads, and budgets, guardrails and PII redaction apply before the spend happens. The pay-as-you-go page lists 500 models from 30 providers, native OpenTelemetry support and a 30-day retention window.
- Pricing: Usage-based. Pay-as-you-go charges a 4.5% fee on spend through Orq.ai keys, with 1M free requests a month on your own keys. Free plan: free plan.
- Best for: European teams wanting a governed model gateway
- Strengths: OpenAI-compatible gateway; 500+ models from 30+ providers; MCP gateway.
Where it falls short
The center of gravity is the gateway, so evaluation depth and dataset tooling are lighter than at Braintrust or Langfuse. Routing every call through a third party adds a dependency to the request path.
Wrong for
A team that only wants to record traces from code it already runs and has no need for a gateway.
Visit Orq.ai →
#9 Portkey
AI gateway with logs, traces and prompt management · United States · portkey.ai
Gateway Guardrails Prompt management
Portkey is ninth. It is a good choice when a platform team wants every model call in the company to pass through one controlled layer, where keys, budgets, retries and logs live together. The Developer plan is free for 10k recorded logs, and Production at $49 a month adds 100k logs, alerts, guardrails and role-based access.
- Pricing: Usage-based. Developer is free for 10k recorded logs. Free plan: free plan.
- Best for: Platform teams standardizing model access
- Strengths: Universal API across providers; Fallbacks and load balancing; Logs and traces.
Where it falls short
Evaluation is not the headline, so teams that want datasets, judges and experiment comparison will pair it with another tool.
Wrong for
A team that needs deep offline evaluation, annotation queues and a test-dataset workbench; Braintrust, Langfuse or Confident AI cover that job directly.
Visit Portkey →
#10 HoneyHive
Agent tracing and evals for enterprise rollouts · United States · honeyhive.ai
Evals Enterprise Hybrid hosting
HoneyHive ranks tenth because it looks strongest at the enterprise end. The free Developer plan is generous in features, with automated and human evaluation, prompt versioning and CI/CD integration, but limited to 10,000 events and 30 days.
- Pricing: Quote only. Developer is free with 10,000 events a month, 5 users and 30-day retention; Enterprise has custom pricing and usage limits. Free plan: free plan.
- Best for: Enterprises scaling agents in production
- Strengths: Distributed tracing; Alerts and drift detection; Custom dashboards.
Where it falls short
There is no published self-serve paid tier between free and Enterprise, so a team that outgrows 10,000 events must talk to sales to learn the price.
Wrong for
A startup that needs a transparent monthly price at 100k events, or a team that wants an open-source codebase it can fork, should look at Langfuse, Opik or Braintrust.
Visit HoneyHive →
#11 Confident AI
Eval platform built around the DeepEval framework · United States · confident-ai.com
DeepEval CI evals Red teaming
Confident AI earns eleventh place as a specialist. Its open-source DeepEval framework, Apache-2.0 licensed with more than 19,000 GitHub stars, lets engineers write evaluation tests that run like unit tests, and the cloud platform stores the results, traces and datasets. The pricing page states tracing from $1 per GB-month with adjustable retention.
- Pricing: Usage-based. Free has 2 seats, 1 project and 1 GB-month of traces. Free plan: free plan.
- Best for: Engineers who want LLM unit tests in CI
- Strengths: LLM unit and regression testing; Evals in CI/CD; LLM tracing.
Where it falls short
The step from Free to Starter is $200 a month, and Free allows two seats, one project and five test runs a week, which blocks real evaluation. Team costs $2,000 a month, a large jump.
Wrong for
A team that mostly needs production monitoring, a gateway or a prompt registry, or one that wants a gentle price slope between a free tier and the first paid plan.
Visit Confident AI →
#12 PromptLayer
Prompt registry and request logs for mixed teams · United States · promptlayer.com
Prompt registry Visual evals EU cloud option
PromptLayer comes last in this ranking by merit for observability, but it fits a specific workplace: a company where a support manager or domain expert edits prompts and nobody wants to deploy code for every wording change. Prompts are versioned and released from the registry, requests are logged, and evaluation cells run against datasets.
- Pricing: Flat monthly + usage. Free has 5 users and 2.5k requests a month. Free plan: free plan.
- Best for: Teams where product staff edit prompts
- Strengths: Prompt versioning; Request logs; Playground.
Where it falls short
Free, Pro and Team are hosted in the US, and EU hosting needs Enterprise, which matters to European buyers. Role-based access control, SSO and deployment approvals are Enterprise-only, and Team still has basic support.
Wrong for
An engineering team that wants deep agent traces, OpenTelemetry ingestion and an open-source server; Langfuse, Opik or Phoenix are closer to that need.
Visit PromptLayer →
Questions and answers
What is the best LLM observability and evaluation tool in 2026?
Langfuse ranks first of 12, best for teams wanting open source plus managed cloud. Opik is second (teams that must self-host the whole stack) and Pydantic Logfire third (python teams already on Pydantic AI). Places 1 to 3 go to challengers, vendors outside the best-known brands, wherever the category has them.
What is the difference between LLM observability and LLM evaluation?
Observability records what happened on each request: the prompt, the retrieved context, the tool calls, the answer, the latency and the cost. Evaluation judges whether the answer was good, using an LLM judge, a code check or a human reviewer.
Do I need an LLM observability tool if I already use Datadog or another APM?
An APM shows latency, errors and infrastructure, but it does not know what a prompt, a judge score or a dataset is. Datadog has added LLM spans, so teams on it can start there and keep traces beside application data.
Can I self-host these tools?
Yes for several. Opik's full platform is Apache-2.0 and free to self-host. Langfuse publishes Docker Compose, Kubernetes and Terraform guides. Phoenix runs locally or in your environment under the Elastic License 2.0.
How are LLM observability tools priced?
By a usage unit that differs per vendor: spans (Opik, Phoenix, Datadog), events (HoneyHive), recorded logs (Portkey), processed gigabytes plus scores (Braintrust), stored trace gigabyte-months (Confident AI) or standard units (LangSmith).
Does tracing add latency to my application?
SDK-based tracing sends spans asynchronously in batches, so the added latency on the request path is small, but you should verify it with your own load test and set sensible flush timeouts.
Which tools work with OpenTelemetry?
Langfuse, Opik, Phoenix, Logfire and Orq.ai each state OpenTelemetry support on their own pages.
Can the data be used for fine-tuning or training?
The stored traces and corrected outputs are often the best source of training and test data, which is why export matters.
What is an LLM-as-a-judge evaluator and how reliable is it?
It is a second model prompted to score the first model's output against a rubric, such as faithfulness to the retrieved context or tone.
How long should I keep traces?
Keep full traces as long as you need to reproduce an incident or an audit question, and keep a curated dataset indefinitely.
Is there an open-source alternative to a paid platform?
Yes. Opik is Apache-2.0, Langfuse's core is MIT, Phoenix is source-available under the Elastic License 2.0, and Logfire and LangSmith publish open SDKs under MIT. DeepEval from Confident AI is an Apache-2.0 evaluation framework.
