LangSmith vs. ClawMetry: an honest comparison for AI agent observability

“We’re a LangChain shop: should we just use LangSmith?” I get this question every week. The short answer is: it depends on what you’re trying to observe. The longer answer is this post.

I built ClawMetry, so I have a stake in this. I’ve tried to be accurate rather than promotional. If anything reads as spin, call it out on GitHub.

What each tool actually is

LangSmith is LangChain, Inc.’s commercial observability and evaluation platform. Its core abstraction is the run (equivalent to a trace): every LLM call, chain execution, tool invocation, or retrieval step is a run with a parent. On top of that you get a prompt playground, datasets for regression testing, an evaluation framework, and a feedback annotation UI. It integrates natively with LangChain and LangGraph, and via a thin SDK decorator with anything else.

ClawMetry is an open-source observability dashboard for AI agents across 14 runtimes: OpenClaw, Claude Code, Codex, NVIDIA NeMoClaw, and 10 more. Its core abstraction is the agent session: what tasks the agent worked on, which tools it called, what sub-agents it spawned, what cron jobs ran, and how memory files changed. Zero instrumentation required for supported runtimes: ClawMetry auto-detects by reading session files and connecting to the gateway WebSocket.

That difference cascades into almost every other dimension of this comparison: LangSmith is an instrumentation-and-eval platform; ClawMetry is an agent operations platform. They solve adjacent problems from different starting assumptions.

Quick comparison

The OSS gap: this one matters more than people admit

LangSmith’s SDK is open-source (MIT). The server (the thing that stores your traces, runs your evals, and hosts the UI) is not. That means if LangChain, Inc. changes pricing, shuts down, or acquires a competitor, your self-hosted option is a Docker image of a closed binary with no ability to audit what it does with your data.

ClawMetry is fully open source. The sync daemon (clawmetry/sync.py), the DuckDB store (clawmetry/local_store.py), the dashboard (dashboard.py), the route modules: all of it is on GitHub under MIT. You can fork it, host it, audit the encryption code, verify that the E2E-encrypted cloud sync actually can’t be read server-side. There is no trust-me-bro layer.

For teams that have made a deliberate choice to run open-source infrastructure, this is a genuine architectural difference, not a marketing claim.

Pricing model

LangSmith’s free tier covers 5,000 traces per month. A “trace” in LangSmith is one top-level run: a chain execution, an agent step, or a single LLM call if you instrument at that level. A moderately busy agent doing 50 LLM calls per task and 200 tasks per day generates roughly 200 top-level traces and 10,000 child runs. Whether that counts as 200 or 10,200 traces against your quota depends on how you instrument it. Either way, you hit the paid tier fast on any production workload.

The Developer plan at ~$39/month includes more traces with per-event overages above the limit. Enterprise pricing is negotiated. The pattern is familiar: it scales with usage, which is fine for low-frequency LLM apps and punishing for high-frequency agent loops.

ClawMetry OSS is free forever. There is no trace quota, no per-event meter, no usage ceiling on the open-source stack. Cloud-Pro adds the features that genuinely require a server: multi-node fleet, alert rules, Slack/PagerDuty webhooks, and retention beyond 24 hours. The separation is deliberate: the local observability loop should never cost money.

Data residency

This is the dimension I get the most pointed questions about from enterprise teams.

LangSmith Cloud sends your trace data (prompt content, model responses, tool call inputs/outputs, token counts) to LangChain, Inc.’s servers. Their privacy policy governs what happens to it. For many teams building internal tools or processing sensitive documents, that is a non-starter.

LangSmith self-host is an option, but it requires running a Docker Compose stack with Postgres, Redis, and the closed-source LangSmith server binary. You own the infrastructure, but you’re still running software you can’t audit.

ClawMetry is local-first. The sync daemon reads OpenClaw’s filesystem and stores everything in a DuckDB file on the agent’s machine. Nothing leaves your environment by default. When you opt into cloud, the snapshot is AES-256-GCM encrypted client-side before transmission: we can’t read it, only your browser can decrypt it. The encryption code is in clawmetry/sync.py on GitHub; you can audit it.

Agent specificity: runs vs. sessions

LangSmith’s data model is built around runs. A run is a single invocation of a traceable unit: one LLM call, one tool, one chain. LangGraph adds some agent-specific shape: you can see the graph node that produced each run, but the fundamental unit is still a discrete invocation.

Agent observability is a different problem. When an OpenClaw agent spawns three sub-agents to parallelize a research task overnight, you don’t just want a flat list of 80 LLM calls. You want:

• Which sub-agent made which calls, and what was its total cost?
• Which one failed silently at 2 AM with zero retries?
• Did the cron job that triggered all of this run on schedule?
• What changed in MEMORY.md after the session ended?

ClawMetry’s data model is built around these concepts from the start. A session is the top-level unit. Inside it you see tool calls with latency and outcome, sub-agents with their own sessions nested inside, cron jobs with schedule adherence and failure history, and memory diffs showing exactly which lines changed in which files. These don’t map cleanly onto LangSmith’s run tree model because they were never designed to.

ClawMetry sub-agent tree: each node is a full session with its own cost, duration, and outcome. LangSmith traces individual runs inside those nodes; ClawMetry connects them into an agent operations view.

Neither model is wrong. They reflect different frames: LangSmith asks “what did each LLM call do?” ClawMetry asks “what did the agent accomplish, what did it cost, and what broke?”

Time-to-first-insight

LangSmith instrumentation for a LangChain/LangGraph stack takes about 10 minutes: set LANGCHAIN_API_KEY, enable tracing in your chain config, and runs start flowing. For non-LangChain stacks, you use the @traceable decorator to wrap your functions, still fast for simple pipelines, but genuinely tedious for complex agent code where you want every sub-agent, tool, and retry branch to show up correctly.

ClawMetry on any of its 14 supported runtimes:

pip install clawmetry && clawmetry

No config file. No API key. No code changes. ClawMetry reads ~/.openclaw/agents/main/sessions/*.jsonl, connects to the gateway WebSocket, and has a live dashboard in under 60 seconds. The zero-config design is possible because we target specific runtimes: each one gets a purpose-built auto-detection path rather than a generic SDK hook.

The LangChain ecosystem bet

LangSmith’s deepest moat is its native integration with LangChain and LangGraph. If your agents are built on LangGraph, LangSmith gives you something ClawMetry doesn’t: first-class graph state visualization, step-through debugging of individual graph edges, and the LangGraph Studio IDE integration. That’s a real advantage for teams who have made a LangGraph architectural bet.

The flip side: LangSmith is architected for LangChain’s framework, which means its data model and UI are optimized for LangChain-shaped workloads. If you move away from LangChain, you’re re-instrumenting your observability layer from scratch. ClawMetry’s auto-detection is tied to runtime binaries (OpenClaw, Claude Code, etc.) rather than to a framework, so it survives framework migrations.

When to pick LangSmith (not us)

I want to be direct here. These are genuine cases where I’d tell you to use LangSmith:

• You’re building on LangChain or LangGraph. The native integration is unbeatable. Zero friction to get traces. LangGraph Studio and the graph state debugger are real productivity multipliers for LangGraph development. If your team is committing to the LangChain ecosystem, LangSmith is the obvious observability layer.
• You need prompt management and A/B testing. LangSmith lets you version prompts, run them against datasets, compare outputs, and deploy with a click. ClawMetry has no prompt management and no plans to build it. If prompt iteration speed is the bottleneck, LangSmith is the right tool.
• You need evaluation pipelines. LLM-as-a-judge scoring, human annotation queues, regression datasets, custom evaluators: LangSmith has all of it. If you’re doing serious quality measurement work on LLM outputs, ClawMetry has nothing here.
• Your stack isn’t on ClawMetry’s supported runtime list. ClawMetry auto-detects 14 runtimes today: OpenClaw, Claude Code, Codex, NVIDIA NeMoClaw, and 10 others. If your agent runs on a custom framework, AutoGen, CrewAI, or a stack not on that list, ClawMetry’s zero-config auto-detection won’t work. LangSmith’s @traceable decorator is the right answer for custom stacks.
• Cross-provider LLM benchmarking is important to you. You’re running experiments across GPT-4o, Claude, Gemini, and local models and want unified latency and cost comparisons. LangSmith has integrations for all of them. ClawMetry surfaces whatever model the OpenClaw gateway routes to but doesn’t give you cross-provider benchmarks.

When to pick ClawMetry

• You run any of the 14 supported runtimes. Zero instrumentation, zero code changes, live dashboard in 60 seconds. No other tool has this for OpenClaw, Claude Code, Codex, and NVIDIA NeMoClaw.
• Data residency is non-negotiable. Local-first means your agent’s prompt content, memory files, and tool call history never leave your machine unless you opt in to encrypted cloud sync. Fully open-source means you can verify that claim by reading the code.
• You need agent operations visibility, not just LLM tracing. Cron job health, memory file diffs, sub-agent cost attribution, multi-node fleet view, channel adapter status (Telegram, Discord, Slack, WhatsApp…): these are ClawMetry-native. You can reconstruct some of this from LangSmith runs with careful instrumentation, but you’d be building ClawMetry on top of LangSmith by hand.
• You want a completely open-source stack. The server, the store, the sync daemon, the dashboard: everything is MIT-licensed on GitHub. No trust-me-bro components, no closed binary in your Docker Compose.
• You run agents on multiple machines. ClawMetry’s multi-node fleet view aggregates all nodes into a single dashboard. LangSmith’s project structure lets you filter runs by project but doesn’t have a first-class “fleet of agents across machines” concept.

Using both

Some teams do this and it makes sense. LangSmith handles LLM call tracing and evaluation at the model layer (latency per model, token cost per call, output quality scoring). ClawMetry handles agent session observability at the orchestration layer (which sub-agent ran, what cron jobs fired, what memory changed, what the fleet-wide cost was).

They operate at different levels of abstraction and don’t conflict. LangSmith tells you what the GPT-4o call inside sub-agent #3 returned. ClawMetry tells you that sub-agent #3 was spawned by the overnight research task, ran for 47 minutes, and cost $1.84 across all its model calls combined.

Bottom line

I built ClawMetry because I was running OpenClaw agents and had no visibility into what they were doing. Token costs were a black box. Cron jobs failed silently. Sub-agent status required tailing multiple log files simultaneously. pip install clawmetry was the tool I wished existed.

LangSmith is a genuinely strong product for LangChain teams. The eval capabilities in particular have real depth: dataset management, LLM-as-a-judge scoring, and annotation queues are things ClawMetry doesn’t have and isn’t trying to build.

The decision is really about what you’re measuring. If you’re measuring LLM call quality across providers and iterations, LangSmith wins. If you’re measuring whether your agent fleet is healthy, what it costs to run overnight, and why that cron job didn’t fire, with no data leaving your machine, that’s ClawMetry.