Arize Phoenix vs. ClawMetry: an honest comparison for AI agent observability

I keep seeing Arize Phoenix in the same breath as ClawMetry in agent tooling discussions, and for good reason. Phoenix is one of the few tools in this space that is genuinely OSS all the way down, and its instrumentation model is legitimately thoughtful. So let me do what I tried to do with the LangSmith and Langfuse posts: compare them honestly, including the cases where Phoenix wins.

I built ClawMetry, so I have a stake in this comparison. If anything reads as spin, the right place to call it out is GitHub.

What each tool actually is

Arize Phoenix is the open-source observability and evaluation platform from Arize AI. Its foundation is OpenInference, Arize’s extension of OpenTelemetry semantic conventions for AI workloads. You instrument your code with one of its framework-specific packages (openinference-instrumentation-langchain, openinference-instrumentation-openai, openinference-instrumentation-llama-index, and a dozen more), and Phoenix collects OTel spans into a local server that renders a trace UI, a dataset manager, and an evaluation harness. It can run embedded in a notebook (import phoenix as px; px.launch_app()) or as a standalone Docker container.

ClawMetry is an open-source observability dashboard for AI agents. Per ENTITLEMENTS.md, it auto-detects agent runtimes with OpenClaw and NVIDIA NeMoClaw supported free in the open-source app, and Claude Code, Codex, Cursor, Goose, and more available via Cloud or Pro tier. Zero instrumentation required: ClawMetry auto-detects by reading session files from ~/.openclaw/ and connecting to the OpenClaw gateway WebSocket. Its core abstraction is the agent session: what tasks the agent worked on, which tools it called, what sub-agents it spawned, what cron jobs ran, and what changed in memory files. It also accepts OTLP ingest on /v1/metrics, /v1/traces, and /v1/logs, so it can receive Phoenix-style OTel spans from runtimes it doesn’t natively auto-detect.

The key fork in philosophy: Phoenix is an instrumentation-first platform built on OTel. ClawMetry is an agent-runtime-first platform built on auto-detection. They observe AI workloads from fundamentally different entry points.

Quick comparison

OSS posture: both genuinely open, different scopes

Both tools are real open source. This is rarer than it sounds in the observability space: several competitors ship a closed server behind an open SDK. Phoenix is Apache 2.0 on GitHub, server included. ClawMetry is MIT on GitHub, same story. If OSS posture is your selection criterion, both clear the bar.

The difference is in the open-core business model. Arize AI’s commercial product (Arize Platform) is a separate, closed-source SaaS product that competes with commercial APM tools at the enterprise layer. Phoenix is Arize’s OSS lead-gen, not the enterprise product itself. That’s a legitimate model, and I’m not criticizing it, but it means Phoenix’s development roadmap is partly driven by what features will pull users toward Arize Platform.

ClawMetry’s open-core split is different: the free tier IS the full local observability product. Cloud-Pro adds the features that genuinely require a server: multi-node fleet aggregation, alert webhooks, retention beyond 24 hours. Nothing in the local product is paywalled or crippled to nudge you to cloud.

Instrumentation vs. zero-config

Phoenix’s instrumentation model is one of the cleaner designs I’ve seen. You pick your framework and install its OpenInference package:

pip install openinference-instrumentation-langchain

# Then in your code:
from openinference.instrumentation.langchain import LangChainInstrumentor
LangChainInstrumentor().instrument()

That’s it. Phoenix auto-registers as the OTel backend and starts collecting spans. For LangChain, LlamaIndex, and a handful of other frameworks, the instrumentor does genuinely smart work: it traces each chain step, captures prompt and response text, records token counts, and links retrieval queries to their documents. For frameworks without an instrumentor, you fall back to the OTel SDK and decorate your functions manually, which is more work but standard.

ClawMetry’s approach: don’t instrument at all.

pip install clawmetry && clawmetry

No imports, no decorators, no code changes. ClawMetry reads ~/.openclaw/agents/main/sessions/*.jsonl for OpenClaw sessions, connects to the gateway WebSocket for live data, and opens a live dashboard in under 60 seconds. This is possible because we target specific runtimes: each gets a purpose-built auto-detection path instead of a generic SDK hook.

The trade-off is real. Phoenix’s SDK works with any framework that has Python code you can modify. ClawMetry’s zero-config only works if your agent runs one of the runtimes on the supported list. Outside that list, you either instrument with OTel and send the spans to ClawMetry’s OTLP endpoint, or you use Phoenix.

Data models: spans vs. sessions

Phoenix inherits OTel’s span-and-trace hierarchy, extended with AI-specific attributes from the OpenInference spec: llm.token_count.prompt, llm.model_name, retrieval.documents, embedding.text. A trace is one request through your system; each span is one step. It’s a horizontal slice through time.

The OpenInference conventions are well-thought-out. The attributes are standardized enough that Phoenix can run evaluations on any span from any framework without knowing which framework produced it. That generality is valuable if you’re mixing LangChain and raw OpenAI calls in the same system.

ClawMetry’s data model is built around a different question. A span tells you what happened inside one request. A ClawMetry session tells you what an agent accomplished over hours or days:

• Which tasks did the agent attempt, complete, or abandon?
• What did each sub-agent cost, and did it succeed?
• Which cron job triggered this session, and did it fire on schedule?
• What changed in MEMORY.md after the session ended?
• How does this session compare to yesterday’s across the whole fleet?

These are the questions that wake people up at 3 AM when an agent pipeline has been running overnight. Phoenix can reconstruct some of them from span attributes if you instrument carefully, but you’re hand-building the ClawMetry session model on top of the Phoenix trace model. That’s a lot of instrumentation work for concepts that should be first-class.

Phoenix excels at the per-request view: every LLM call, retrieval step, and token count, rendered as a trace. ClawMetry’s session view asks the question one level up: what did the agent accomplish, and is the fleet healthy?

RAG pipelines and evaluation: Phoenix’s genuine strength

I want to be direct about where Phoenix is the better tool, because it’s a real advantage, not a quibble.

If you’re building a RAG pipeline (query → retrieval → rerank → generation), Phoenix has first-class support for every step. Retrieval spans carry the query, the retrieved documents, their scores, and latency. The LlamaIndex instrumentor is particularly thorough: it traces every node in a query engine, including intermediate reasoning steps in agentic RAG.

Beyond tracing, Phoenix ships with evaluators you can run inline:

from phoenix.evals import HallucinationEvaluator, QAEvaluator
from phoenix.evals import run_evals

hallucination_eval = HallucinationEvaluator(model=eval_model)
qa_eval = QAEvaluator(model=eval_model)
results = run_evals(dataframe=traces_df, evaluators=[hallucination_eval, qa_eval])

You get a DataFrame back with per-trace hallucination scores and Q&A correctness labels. You can export failing examples to a dataset, version the dataset, and run regression checks on the next model or prompt change. This is a genuine eval loop, not a mock-up.

ClawMetry has none of this. I’m not building eval pipelines. That’s a deliberate scope decision: ClawMetry observes operational health, not output quality. If your job is shipping a better RAG system and you need to measure whether answers improved after a retrieval tweak, Phoenix is the right tool.

Agent operations: where ClawMetry goes further

The story flips for long-running agent operations. Phoenix sees what happened inside each LLM call. It doesn’t see whether your overnight research agent’s cron job fired, which sub-agent ran over budget and got killed, or what changed in the agent’s persistent memory between Monday and Tuesday.

ClawMetry’s cron tracker, for example, shows every job with its schedule, last-run time, whether it ran late or failed silently, and the full run log. When a cron job stops firing (not an error, just silence), Phoenix has no way to surface that because it only sees spans that were actually emitted. ClawMetry checks schedule adherence independently of whether the process ran at all.

Memory file diffs are another ClawMetry-native concept. OpenClaw agents write to ~/.openclaw/agents/main/MEMORY.md and similar files to persist state across sessions. ClawMetry diffs those files after every session and shows exactly which lines changed, added, or disappeared. This is the kind of operational visibility you need when an agent is autonomously updating its own task list and you want to audit what it decided to remember.

Multi-node fleet is the third gap. ClawMetry’s fleet view aggregates session counts, token spend, error rates, and cron health across every node in a single dashboard. Phoenix’s self-hosted model runs one server per environment; there is no native fleet aggregation layer.

Data residency: both local-first, different cloud models

Both tools run locally by default, which matters more than it used to. After a run of high-profile data incidents at SaaS observability vendors, enterprise teams are scrutinizing where LLM prompt content actually goes.

Phoenix’s default mode, embedded in-process or as a Docker container, keeps all data on your machine. Nothing leaves until you connect to Arize Platform. The Phoenix server itself stores spans in SQLite or Postgres, both standard, auditable storage layers.

ClawMetry is also local-first. The DuckDB store lives on the agent’s machine. Cloud sync is opt-in, and when you opt in, the snapshot is AES-256-GCM encrypted client-side before it leaves your machine: the ClawMetry server receives ciphertext and a nonce; the decryption key never leaves your browser. That design is documented in clawmetry/sync.py on GitHub, so you can read the implementation, not just the marketing claim.

For teams that need a signed attestation that LLM prompt content never left their perimeter, ClawMetry’s E2E-encrypted design is verifiable. Phoenix’s local-only mode achieves the same guarantee if you never connect to Arize Platform, but there’s no encrypted path that gives you both cloud access and data sovereignty.

Time-to-first-insight in practice

Phoenix is genuinely fast for supported frameworks. Starting from zero on a LangChain project:

pip install arize-phoenix openinference-instrumentation-langchain

# In your script:
import phoenix as px
px.launch_app()

from openinference.instrumentation.langchain import LangChainInstrumentor
LangChainInstrumentor().instrument()

# Your existing LangChain code runs unchanged below this point

From zero to first trace in a browser: about 5 minutes. That’s honest. For custom stacks without a pre-built instrumentor, you’re adding OTel spans manually around your LLM calls, 30–60 minutes to do it properly.

ClawMetry on an OpenClaw, Claude Code, or Codex workspace:

pip install clawmetry && clawmetry

First session appears in the dashboard in under 60 seconds. No modification to existing code. If your runtime isn’t on the supported list, time-to-first-insight depends on how long it takes to add OTel instrumentation, same as Phoenix.

When to pick Arize Phoenix (not us)

I want this section to be usable, not a fig leaf. These are genuine cases where I’d recommend Phoenix:

• You’re building a RAG pipeline. Phoenix’s retrieval span model and inline hallucination/relevance evaluators are the best in the OSS space for this use case. If your core loop is query → retrieve → generate and you need to measure whether retrieval quality improved, Phoenix is the right tool.
• You need LLM output evaluation. Phoenix ships real evaluators: hallucination detection, Q&A correctness, toxicity, SQL correctness. You can run regression checks across datasets after each prompt change. ClawMetry has nothing here and no plans to build it.
• Your stack isn’t on ClawMetry’s supported runtime list. If you’re running LlamaIndex, DSPy, AutoGen, CrewAI, or a custom agentic framework, Phoenix’s instrumentation approach covers you. ClawMetry’s zero-config only works for runtimes it natively auto-detects.
• You want cross-provider LLM benchmarking. OpenInference instrumentors exist for OpenAI, Anthropic, Google Gemini, Mistral, AWS Bedrock, and more. Phoenix gives you a unified trace view across providers in a single dashboard, useful for A/B testing models.
• You’re already running OTel infrastructure. If your platform team has Jaeger, Tempo, or another OTel collector deployed, Phoenix’s spans drop straight in. You get AI-specific visualization on top of existing observability infrastructure without a new data silo.
• You prefer PostgreSQL for storage. Phoenix supports Postgres as its backend store in Docker mode, which slots into standard DBA workflows for backup, replication, and access control. DuckDB is great for analytics; it’s unfamiliar to most DBAs.

When to pick ClawMetry

• You run one of ClawMetry’s supported runtimes. Zero code changes. Live dashboard in 60 seconds. No other tool gives you this for OpenClaw, Claude Code, Codex, and NVIDIA NeMoClaw.
• You need agent operations visibility, not LLM call tracing. Cron job health, memory file diffs, sub-agent cost attribution, multi-node fleet aggregation, and 23 channel adapter statuses per ENTITLEMENTS.md (Telegram, Discord, Slack, WhatsApp, iMessage…): these are ClawMetry-native. Phoenix doesn’t model them.
• Data sovereignty with cloud access matters. ClawMetry’s E2E-encrypted cloud sync means you get a hosted dashboard without your prompt content ever being decryptable by ClawMetry servers. The encryption code is auditable on GitHub.
• You want a complete OSS stack with no Postgres or Redis dependency. ClawMetry runs on DuckDB. One embedded file, zero external services, zero ops overhead.
• You run agents on multiple machines. ClawMetry’s fleet view aggregates all nodes into one dashboard. Phoenix’s self-hosted model runs one server per environment; fleet aggregation requires building your own aggregation layer on top.

Using both

I’ve seen teams run Phoenix and ClawMetry together, and it makes sense at different levels of the stack. Phoenix handles per-LLM-call visibility: prompt content, token counts, model latency, output quality scores. ClawMetry handles session-level and fleet-level visibility: which agent ran, what it cost, whether the cron fired, what changed in memory.

If your agents produce OTel spans (for example, from a Phoenix instrumentor), you can also forward those spans to ClawMetry’s OTLP endpoint at /v1/traces. ClawMetry stores them in DuckDB and you get both views in one dashboard without running two servers. That’s an uncommon setup today but it’s there if you want it.

Bottom line

Phoenix and ClawMetry are both genuinely open source and both run locally by default. Those are real shared values. The split is in what they’re optimizing for.

Phoenix is optimized for making every LLM call legible: what went in, what came out, was it hallucinated, did retrieval help. It’s instrumentation-first and evaluation-first. If your daily question is “is my RAG pipeline producing accurate answers and how do I improve it,” Phoenix is the sharper tool.

ClawMetry is optimized for making AI agents legible as operational systems: are they running, are the cron jobs on schedule, what did the overnight run cost, what changed in memory, is the fleet healthy. It’s runtime-first and zero-config-first. If your daily question is “what did my agent fleet do last night and what broke,” that’s ClawMetry.

Both tools are worth knowing. They’re mostly additive, not substitutes.