Claw
IBM Technology’s explainer “Are Your AI Agents Flying Blind? The Truth About AgentOps” does something most AgentOps content skips: it names the metrics. Three layers, observability, evaluation and optimization, with the numbers that matter in each, walked through on a two-agent prior-authorization system.
That makes it a usable checklist. So I scored ClawMetry against it, one metric at a time, using the code that ships today rather than the roadmap. I used three verdicts:
- Alerts: ClawMetry computes it and can notify you, or act, when it goes wrong.
- Shows: you can see it in the dashboard, but nothing fires when it drifts.
- Missing: not measured.
The short version: 3 alert, 7 show, 5 are missing. The pattern behind that split says as much about the category as it does about us.
Layer 1: observability, “what happened?”
| Metric | Verdict | What ClawMetry has |
|---|---|---|
| Cost per request | Alerts | Cost per session and per completed task from a multi-provider pricing table; a dollars-per-completed-task alert rule; budget alerts. |
| End-to-end trace duration | Shows | Per-trace duration in Tracing. An SLA panel shows p95 completion time against a target, green or red, but it never sends an alert. |
| Tool execution latency | Shows | Per-tool call counts, p50/p95 latency and error rate. No latency alert, and no calls-per-request figure. |
| Agent-to-agent handoff latency | Missing | Sub-agent spawns are recorded as spans with start and end times, but no handoff-latency metric is computed from them. |
Cost is where ClawMetry started, and it shows. Latency is the opposite. We collect the timings and never turn them into an SLO that pages someone. The SLA panel going red on a screen nobody is looking at is exactly what the video calls flying blind.
Layer 2: evaluation, “was it any good?”
| Metric | Verdict | What ClawMetry has |
|---|---|---|
| Task completion rate | Alerts | Every session gets an outcome label (success, failed, escalated, stuck in a loop, stuck on a tool call, and so on); an outcome-failure-rate alert rule. The needed-a-human rate is computed but cannot be alerted on on its own. |
| Guardrail violation rate | Shows | Every Guard policy action and every pre-tool approval is logged with an audit trail. You get the list and a count, not a rate, and no alert on it. |
| Human review sampling | Shows | A review queue that samples sessions for a person to mark correct, wrong or borderline, with an accuracy figure. It samples a fixed count per agent per day (10 by default), not the video’s 5%, and does not alert when accuracy drops. |
| Factual accuracy vs source records | Missing | A faithfulness score (Pro) checks claims against evidence in the transcript. Nothing compares output to the system of record. |
| First-pass approval rate | Missing | Outcomes are joined to git and pull requests for coding agents. There is no way to post a business outcome such as “the payer approved it”. |
This is the layer where an observer runs into its own limits. ClawMetry reads what the agent did from the outside. It can tell that a session ended cleanly and that the final answer is consistent with what the agent read. It cannot tell that the diagnosis code was right, because “right” is defined by a record it has never seen. The two missing rows are the same missing feature: a place for your system to tell ClawMetry what actually happened.
Layer 3: optimization, “how do we make it better?”
| Metric | Verdict | What ClawMetry has |
|---|---|---|
| Infinite loops (the budget-burn case) | Alerts | Loop and no-progress detectors with learned per-runtime thresholds, a spend-at-risk figure on every incident, and Guard policies that can pause and then kill, all opt-in. |
| Prompt token efficiency | Shows | Token and cache analytics, plus a cohort comparison by model, runtime version, repo and date. A prompt change is not a cohort key yet, so “same quality with 40% fewer tokens” has to be compared by date range. |
| Flow step efficiency | Shows | Steps per session in cohort comparison. No notion of the optimal path, so there is no “7.2 steps vs 6” overhead figure. |
| Handoff success rate | Shows | The sub-agent tree and child outcomes are visible per session; nothing rolls them up into a rate. |
| Retrieval precision@K | Missing | Retrieval spans are labelled in traces, but not scored. |
| Improvement velocity | Missing | Not in the open-source build; the self-improvement loop is a Pro feature, and nothing counts shipped optimizations or their measured effect. |
(I added the loop row back into layer 3 because it is where the video’s budget worry belongs operationally. The video itself lists nine metrics across the three layers plus a handful more in the worked example. Fifteen is the union.)
The pattern
Line the verdicts up and a shape appears. Everything ClawMetry alerts on is a property of the runtime: cost, loops, failures, completion. Everything it only shows is a property of performance over time: latency, rates, drift. Everything missing is a property of correctness in your domain: accuracy, approval, retrieval quality.
That is not an accident of our roadmap. It follows how far an observer can see from where it sits:
- Runtime failures are visible from outside. A loop is a loop in any domain.
- Performance drift needs only aggregation and a threshold, which we have the data for and have not built. That is on us.
- Correctness needs ground truth that only you have. The best any observer can do is give you a clean place to put it, then do the arithmetic.
So the middle column is the cheap one to fix, and the right column needs a contract with your system rather than a smarter model.
What we’re building from this
- Latency SLO alerts. A p95 trace-duration and per-tool-latency alert rule, and the SLA panel wired to actually fire.
- Rates you can alert on. Handoff failure rate, human-escalation rate and guardrail violation rate as first-class alert rules next to the existing outcome-failure rate.
- An outcomes endpoint. Post the real result of a session (code correct, payer approved first pass) and get accuracy and first-pass rates as measured numbers.
- Prompt as a cohort key. Fingerprint the system prompt and config per session, so before-and-after on a prompt change is one click, with quality per input token.
- Percentage review sampling, with an alert when reviewer-marked accuracy drops.
There is a sharper version of this audit that follows the video’s own scenario question by question, including the case of a custom agent reporting over OpenTelemetry traces, where the gaps are bigger: Could ClawMetry have caught IBM’s prior-auth agents going wrong?
The honest part
Three of fifteen is not a number a vendor normally publishes. I’m publishing it because the checklist is good, and because a scorecard you can check against the code is worth more to you than a feature grid that says yes to everything. If you think I scored something wrong in either direction, open an issue and I will fix the post.
More from the blog
See what your AI agents are doing
Zero instrumentation. Local-first. Fully open source. 726k+ installs.
Get ClawMetry free