Security and compliance teams asking "which LLM observability tool should we use" are usually asking a different question than the engineering teams who first evaluated these products. Langfuse, LangSmith, Arize Phoenix, and Braintrust were each built primarily to help engineers debug prompts and run evaluations — audit-trail durability, retention guarantees, and compliance-grade access control were not the design center for any of them. This post evaluates the same four tools specifically against that lens. For the general criteria that apply across any agent observability tooling — not just these four named vendors — see evaluating AI agent observability tooling.
Why "built for eval" and "built for audit" are different requirements
An eval-first tool needs to answer: did this prompt version perform better than the last one, and by how much? That calls for fast iteration, dataset management, and scoring — retention of a specific trace for years, immutability guarantees, and access controls scoped for an external auditor are not the same problem, and most tools were not designed against both at once.
A compliance-grade requirement set looks different: durable retention on a schedule you control (not the vendor's default), tamper-evident or append-only storage so a trace cannot be silently edited after the fact, identity attribution on every record, and export paths that produce evidence an auditor will actually accept. See tamper-evident audit logs and cryptographic proofs for what that integrity bar generally requires, and audit trails that hold up for what auditors look for beyond storage alone.
None of the four tools below is disqualified by this distinction — several offer enterprise tiers or self-hosting that can be configured toward compliance needs. The point is that "observability tool" and "audit system" are overlapping but distinct categories, and evaluating one as if it were the other produces the wrong shortlist. The broader case for keeping regulatory-facing evidence architecturally separate from day-to-day telemetry is covered in the AI governance guide.
It is worth being specific about what "compliance readiness" means in practice, because the term gets used loosely. It generally breaks down into four separable properties: retention you control (not a vendor default that can change), integrity (a record that cannot be silently edited after the fact), attribution (which identity performed which action, on every record, not just some), and exportability (evidence in a format a third party — an auditor, a regulator, opposing counsel — will actually accept without needing the vendor's own viewer). A tool can be strong on one property and weak on another; treating "compliance readiness" as a single yes/no checkbox is how teams end up surprised during an actual audit.
The four tools
Langfuse: open-source, prompt-centric
Langfuse is open-source under the MIT license and, as of its integration into ClickHouse, backed by a company with a strong analytics-database pedigree. Its public documentation centers on prompt-centric workflows: tracing, evaluation, and dataset management for iterating on LLM applications. Being open-source and self-hostable gives a compliance team direct control over retention and storage — a genuine advantage for teams that need data to stay in a specific environment — but that control is something you configure and operate yourself; it does not arrive pre-hardened for audit use.
LangSmith: best inside a LangChain or LangGraph stack
LangSmith is closed-source and documented as the natural fit for teams already building on LangChain or LangGraph, where its tracing integrates most tightly. Self-hosting is available only at its Enterprise tier per public documentation, which matters directly for compliance teams: if data residency or on-premises retention is a hard requirement, that requirement gates which pricing tier is even viable, not just which features are available.
Arize Phoenix: OpenTelemetry-native
Arize Phoenix is open-source and built on OpenTelemetry, using the OpenInference semantic conventions for LLM trace data. For a security team already standardized on OTLP export and an existing observability backend (Prometheus, a SIEM, a general-purpose APM), this is a meaningful advantage: trace data is portable to infrastructure you already operate and audit, rather than locked into a vendor-specific schema. See OTLP telemetry ingest and agent discovery and Prometheus metrics and observability for how that portability is typically used downstream. Note that OpenTelemetry's own gen_ai semantic conventions are still unstable as of mid-2026 — see OpenTelemetry GenAI semantic conventions: what's actually stable — so "OpenTelemetry-native" today means aligned with a moving target, not a frozen schema.
Braintrust: eval-first, CI/CD-gated
Braintrust's public positioning is explicitly eval-first: eval scores are native to the trace view, and evaluation results can gate CI/CD pipelines before a prompt or model change ships. That is a strong fit for engineering workflows that want evaluation as a release gate. For a compliance team, the relevant question is narrower: does the CI/CD-gating and scoring history itself produce a durable record suitable as evidence that a change was tested before release — worth confirming directly against current documentation rather than assuming eval history equals audit history.
What to check before shortlisting for compliance
| Question | Why it matters for compliance |
|---|---|
| Can retention be set independently of the vendor default? | Regulatory retention requirements rarely match a vendor's default window |
| Is storage append-only or otherwise tamper-evident? | A trace that can be silently edited is weak evidence in a dispute or audit |
| Is identity attribution (which agent, which caller) captured on every record? | Attribution is what turns a log into evidence, not just a debugging aid |
| Is self-hosting or a compliant deployment region available at the tier you can afford? | Data residency requirements can gate which pricing tier is actually usable |
| Does export produce a format an external auditor will accept? | A proprietary trace viewer is not the same as exportable evidence |
Ask these five questions of each vendor's current documentation before assuming eval-grade tooling meets an audit-grade bar — the answer varies by product and, within a product, often by pricing tier.
It is also worth checking these questions against how each tool handles agent-specific structure, not only single LLM calls. An agent task frequently fans out into a model call plus several tool invocations, and if the observability tool flattens that structure into a single log line, the record loses exactly the decision-path detail an investigator needs later. All four tools above support hierarchical tracing to some degree; how deep that hierarchy goes, and whether tool-call parameters and results are captured alongside the model call, is worth confirming directly rather than assuming from the product name.
Running a compliance-focused evaluation
A proof-of-concept scoped only to engineering fit (does tracing work, is the dashboard usable) will not surface the compliance gaps that matter later. Run the evaluation against your actual retention and evidence requirements instead:
- Configure retention to your regulatory requirement, not the default, and confirm the setting actually holds — some tools apply retention at the project level, others at the organization level, and the two can behave differently under a plan change.
- Attempt to delete or alter a trace as a low-privilege user, and confirm the system either prevents it or produces a durable record that the change occurred. This is the practical test of "tamper-evident," rather than taking the label at face value.
- Export a trace and check whether it is self-contained evidence — does it carry caller identity, timestamps, and full request/response content, or does it require cross-referencing the vendor's own dashboard to be meaningful outside the tool.
- Confirm the pricing tier you can afford actually includes the deployment model you need — several vendors gate self-hosting or specific compliance certifications behind their highest tier, which only becomes visible once you look past the feature list.
Where a dedicated audit layer still matters
Even the strongest eval tool on this list is not purpose-built to be the system of record for a regulator. Most teams that reach real compliance obligations end up running observability tooling (for engineering iteration) alongside a separate audit-grade log (for evidence) rather than treating one as a substitute for the other — the same split discussed generally in observability for AI agents. Praesidia is an AI agent security and governance control plane — agent identity and access, guardrails, audit evidence, and cost controls in one place — built to cover that audit-evidence side of the split described above.
Common questions
Can I just use one of these four tools for both engineering and compliance? Sometimes, if you configure retention, access control, and export deliberately rather than accepting defaults — Langfuse and Arize Phoenix, being self-hostable, give you the most direct control to do this. LangSmith requires its Enterprise tier for self-hosting. Whichever you pick, verify the compliance-relevant settings explicitly rather than assuming the eval-oriented defaults satisfy an audit requirement.
Which of the four is most audit-friendly out of the box? None is purpose-built for it, but open-source, self-hostable options (Langfuse, Arize Phoenix) give a compliance team the most direct control over retention, storage integrity, and export — at the cost of having to configure and operate that hardening yourselves rather than getting it as a managed feature.
Does OpenTelemetry-native mean the trace schema is stable?
No. Being built on OpenTelemetry means trace data is portable and vendor-neutral in structure, which is valuable, but the underlying gen_ai.* semantic conventions themselves are not yet stable as of mid-2026. Portability and stability are separate properties.
Should security teams be involved in choosing an eval tool, or is that purely an engineering decision? Involve security and compliance early if the tool's trace data will ever need to serve as evidence — for an incident investigation, a regulatory inquiry, or a customer audit. Retrofitting retention and access-control requirements onto a tool chosen purely for engineering fit is harder than building the requirement into the initial evaluation.
How does this differ from the general observability evaluation criteria on this site? The general evaluation post covers the criteria that apply to any agent observability tooling — trace granularity, attribution, cost integration, anomaly detection, audit-quality retention — without naming vendors. This post applies those same criteria to four specific, widely used products, with the audit and compliance lens foregrounded rather than eval quality.
Can these tools replace a SIEM or a dedicated audit log? Generally not by themselves. They are built to make LLM application behavior legible to engineers, and several can export data to a SIEM or downstream store, but few are designed from the ground up as the durable, tamper-evident system of record a SIEM or dedicated audit log is expected to be. Treat them as a source that feeds your audit and security infrastructure, covered in general terms in webhooks and SIEM forwarding, rather than as a replacement for it.