Datadog vs. ClawMetry: an honest comparison for AI agent observability

“We already have Datadog for everything else — can we just use it for AI agents too?” This is the most common question I get from platform teams. The short answer is: you can observe your infrastructure around your agents with Datadog, but Datadog LLM Observability and ClawMetry are solving meaningfully different problems. The longer answer is this post.

I built ClawMetry, so I have a stake in this comparison. I’ve tried to be accurate rather than promotional. If anything reads as spin, call it out on GitHub.

What each tool actually is

Datadog is the dominant cloud-era infrastructure monitoring platform: APM traces, logs, metrics, dashboards, alerting, and synthetic testing unified in a single SaaS product. In 2024 they launched Datadog LLM Observability as a dedicated product for tracing LLM calls: prompt and completion content, token counts, latency, model metadata, and error rates. It integrates with OpenAI, Anthropic, and Bedrock via their Python SDK, and surfaces traces in a purpose-built LLM Observability UI alongside the rest of your Datadog stack.

ClawMetry is an open-source observability dashboard purpose-built for AI agent runtimes — 30 of them, from OpenClaw and Claude Code to Codex, NVIDIA NeMoClaw, Cursor, Goose, and more. Its core abstraction is the agent session: what tasks ran, which tools fired, what sub-agents were spawned, what cron jobs executed, what memory files changed, and what the whole tree cost. Zero instrumentation required for supported runtimes: ClawMetry auto-detects supported runtimes and observes them directly, without any code changes or SDK integration.

The fundamental difference: Datadog measures infrastructure and LLM calls; ClawMetry measures agent sessions and their outcomes. Both call themselves “observability,” but they answer different questions.

Quick comparison

The billing model: where Datadog costs compound on agents

Datadog’s pricing is famously complex. For a team adding LLM observability to an existing deployment, the cost picture has multiple layers:

• Infrastructure monitoring: per-host, per-month pricing.
• APM: per-host, per-month. LLM spans appear in APM traces.
• LLM Observability: a separate product with volume-based ingestion pricing.
• Log management: volume-based, per GB ingested and indexed. If your agents generate verbose logs, this adds fast.

The compounding happens at scale. An autonomous agent running overnight might generate thousands of LLM spans, each carrying full prompt and completion content. A fleet of 20 agents doing agentic loops 24/7 can generate millions of spans per month. At volume-based pricing, that is a line item that grows with usage rather than with team size.

ClawMetry stores all session data locally on the agent’s machine. The OSS version has no ingestion quota, no per-span meter, no usage ceiling. You can run 10,000 sessions a day and the storage cost is disk. Cloud-Pro adds server-side features (fleet aggregation, >24h retention, webhooks) on a flat subscription that doesn’t scale with trace volume.

Data residency: this is the deciding factor for many teams

Datadog is cloud-only. Your prompt content — the exact text your agent sends to the LLM — is transmitted to Datadog’s infrastructure to be indexed, stored, and displayed in the LLM Observability UI. For many teams, that is fine. For teams handling customer PII in agent prompts, processing confidential documents, or operating under SOC 2 / GDPR / HIPAA constraints, it is a non-starter.

Datadog offers a “sensitive data scrubber” that can redact patterns from spans before transmission. This helps, but it requires you to enumerate sensitive patterns in advance, and the scrubbing happens after the data is already in the agent buffer, before transmission. One misconfigured pattern and PII reaches Datadog’s servers.

ClawMetry is local-first by design. All agent data is collected and kept locally on the same machine your agents run on — nothing leaves your environment by default. When you opt into cloud sync, your data is AES-256-GCM encrypted client-side before transmission — the server receives ciphertext and cannot read your prompts. The encryption implementation is fully open source on GitHub; you can audit exactly what it does.

The specificity gap: LLM spans vs. agent sessions

Datadog LLM Observability is built on Datadog’s APM span model. An LLM span captures a single model call: input tokens, output tokens, model name, latency, and whether it errored. This is genuinely useful for answering “which model call is the slowest?” and “which prompt is burning tokens?”

Agent observability asks different questions. When an OpenClaw agent spawns three sub-agents to process a research task overnight, you don’t just want a list of 150 LLM spans. You want:

• Which sub-agent produced those 150 spans, and what was the total cost of that branch?
• Did the sub-agent that was supposed to run at 3 AM actually fire? What was the cron’s exit status?
• What changed in RESEARCH_NOTES.md after the session ended, and which specific tool call wrote those lines?
• One sub-agent ran for 4 hours and cost $12. Was that expected? Is it stuck in a loop?

None of these questions are answered by LLM spans. They require an agent-native data model: sessions as first-class entities, sub-agent trees with cost attribution, cron job scheduling and health, memory file diffs, and loop detection. ClawMetry was built around these from day one because they were the questions I couldn’t answer when running OpenClaw agents.

Datadog added LLM Observability as an extension of their existing APM model. It inherits APM’s strengths (deep integration with infra metrics, mature alerting) and its constraints (a span-centric view of the world that flattens agent sessions into flat lists of calls).

ClawMetry’s session view: a sub-agent tree with per-node cost, cron job status, and memory file diffs. Datadog LLM Observability shows the individual LLM spans inside those nodes; ClawMetry connects them into an agent operations picture.

Where Datadog’s unified view genuinely wins

I want to be honest about this: there is a class of problem where Datadog’s unified infrastructure + LLM view is the right answer, and ClawMetry isn’t a substitute.

GPU utilization correlated with LLM latency. Your self-hosted inference stack is occasionally slow. You want to know if it’s model latency, GPU contention, or network. Datadog can show you GPU utilization, CUDA memory, container CPU, and the LLM call latency on the same timeline, correlated. ClawMetry reads agent session data and reports what the agent saw; it doesn’t have visibility into the compute substrate below it.

Kubernetes-native agent deployments. Your agents run as pods in EKS or GKE. You’re already routing everything through Datadog for pod restarts, OOM kills, HPA scaling events, and service mesh traces. Adding LLM Observability means one more product in an already-deployed data pipeline. ClawMetry’s auto-detection is built around locally available agent runtime data; Kubernetes pods with ephemeral storage require explicit configuration.

Incident correlation across services. An LLM-powered feature is degraded. You want to see the spike in p99 latency on the agent service, the upstream database query that slowed down, the Kubernetes node that’s hot, and the LLM call that timed out, all in one correlation view. Datadog’s unified data model makes this possible. ClawMetry is agent-session-scoped; it doesn’t see your database or your Kubernetes cluster.

Enterprise compliance and audit. You need data in Datadog already for compliance reasons. Your SIEM pulls from Datadog. Your SOC team’s runbooks reference Datadog dashboards. Adding a second observability layer creates a gap in the audit trail. The “one pane of glass” argument is real at enterprise scale.

When to pick Datadog (not us)

I’d tell you to use Datadog LLM Observability in these situations:

• You’re already on Datadog and your agents run on infrastructure you monitor there. The marginal cost of adding LLM Observability to an existing Datadog deployment is low: you already have the agent, the API key, the dashboards, and the alerting rules. Adding LLM spans to that pipeline is net-additive.
• You need GPU / container / cluster metrics correlated with LLM calls. Datadog is the only tool that does this well at production scale. If your inference budget and your Kubernetes budget are the same conversation, Datadog keeps it in one place.
• You run agents on frameworks not on ClawMetry’s runtime list. ClawMetry auto-detects 30 runtimes. If your agent is built on AutoGen, CrewAI, custom Python, or a framework we don’t yet support, Datadog’s ddtrace SDK can instrument it with decorator-level granularity.
• Your security team already approved Datadog for sensitive data. Re-litigating data residency for a second observability vendor takes months in an enterprise. If the battle is already won for Datadog, and you have appropriate PII scrubbing in place, that’s a real operational advantage.
• Your primary question is “which prompt version performs better?” Datadog LLM Observability can compare prompt variants in their UI, alongside token cost and latency. ClawMetry has no prompt-management layer. If prompt A/B testing is the workflow, Datadog is built for it.

When to pick ClawMetry

• You run any of the 30 supported runtimes. Zero instrumentation, zero code changes, live dashboard in under 60 seconds. This includes OpenClaw, Claude Code, Codex, NVIDIA NeMoClaw, Goose, Cursor, and 24 more.
• Data residency is non-negotiable. Prompt content, memory files, tool call arguments — none of it leaves your machine by default. Fully open-source means you can audit the claim rather than trusting a privacy policy.
• You need agent-session-level visibility: crons, sub-agents, memory diffs. Cron job scheduling and failure history, sub-agent cost attribution, memory file diffs, loop detection — these are ClawMetry-native. Building them on top of Datadog APM spans is possible in theory; in practice it’s a six-week engineering project that recreates ClawMetry with vendor lock-in.
• You want the observability cost to scale with team size, not with agent activity. Datadog LLM Observability uses volume-based ingestion pricing. As your agents get more capable and run more LLM calls, the bill goes up. ClawMetry OSS has no ingestion meter. Cloud-Pro is a flat subscription.
• You run agents on multiple machines and need a fleet view. ClawMetry’s multi-node fleet view aggregates all nodes into one dashboard with cross-node cost totals. Datadog can show you multi-host metrics, but AI-agent-level fleet aggregation (total session cost across nodes, which node has a stuck sub-agent) requires custom work in Datadog.
• You want a completely open-source stack. MIT license, no closed binary, every component auditable on GitHub. Datadog’s dd-agent is open source; the backend that stores and processes your data is not.

Using both

This combination makes sense for large deployments. Datadog handles the infrastructure and compute layer: GPU utilization, Kubernetes pod health, the database that the agent’s tool queries, the API gateway latency. ClawMetry handles the agent session layer: which sessions ran, what sub-agents they spawned, what cron jobs fired, what memory changed, and what it all cost.

The two tools operate at different levels of abstraction and don’t conflict. Datadog tells you the p95 latency of the Anthropic API call from your LLM inference service. ClawMetry tells you that 40% of that latency is in sessions that are stuck in a tool-call retry loop, and that three of those sessions have been looping for six hours and have spent $34 combined.

If you do run both, route the OTLP traces from your agent’s inference layer to Datadog and let ClawMetry observe the agent runtime directly. They answer different questions without overlap.

Bottom line

I built ClawMetry because I was running OpenClaw agents and had no visibility into what they were doing. Datadog was not the answer — not because it’s bad (it’s not), but because it was not built for the questions I was asking. Why did the cron not fire? Which sub-agent spent $8 overnight? What changed in memory? pip install clawmetry was the tool I wished existed.

Datadog LLM Observability is a genuinely solid product for teams who already live in Datadog and want to extend that investment to LLM call tracing. The unified infra view, the enterprise alerting, and the existing deployment amortize the cost. If that describes your situation, the marginal case for ClawMetry is weaker.

The decision is really about what layer you’re observing. LLM call spans, GPU metrics, and service latency: Datadog. Agent sessions, sub-agent trees, cron health, memory diffs, and agent fleet cost, with no data leaving your machine: ClawMetry.