Estimating the productivity gains from observing your AI agents
If you run AI agents — coding agents in a terminal, support agents on a schedule, a fleet of them while you sleep — you are already paying for them in tokens and in time. The harder question is what you get back from being able to see what they're doing. This is a practical framework for putting a number on that, measured from your own dashboard rather than asserted from ours.
TL;DR. "Productivity gains" from observability isn't one number — it's the sum of waste you stop paying for once you can see it, plus the hours you stop spending digging through logs. We break the dollar side into six measurable categories (reasoning tax, cache misses, silent model fallbacks, tool-failure loops, compaction thrash, runaway loops) and give you the formula to compute each from your own runs. The honest answer to "how much will I save?" is "install it, look, and the dashboard will tell you" — but here's how to reason about it before you do.
Why this is hard to estimate — and why that's the point
Anthropic recently published a careful piece on estimating the productivity gains from AI, and the honest theme is that the gains are real but slippery: they depend on the task, the baseline, and what you actually do with the time you free up. Running agents has the same property, with an extra twist — most of the cost and most of the failure happen where you aren't looking.
An agent spawns sub-agents, re-reads its context every turn, calls tools that sometimes fail, silently falls back to a cheaper model, and compacts its history when it overflows. On a flat-rate or OAuth plan it often reports $0. You see a spinner that says "thinking." You do not see the bill, the loop, or the tool that's failing 40% of the time. You can't estimate — let alone capture — a gain you can't measure. So the first productivity gain of observability is simply turning an unknown into a number.
The dollar side: six categories of waste you can measure
ClawMetry ties every token to the session that spent it and surfaces six waste signals as a chip on each session. Each one is also a line you can put a dollar figure on. Here's how to estimate each from your own runs.
🧠 1. Reasoning tax
Reasoning ("thinking") tokens are billed like output but produce no visible deliverable. On some plans they silently drain a weekly quota. Estimate: for a representative week, take your reasoning-token share of output, multiply by your output rate. If 30% of your output spend is reasoning and you don't need that depth for routine edits, that's a 30%-of-output line you can tune with a cheaper model or a lower effort setting.
⚡ 2. Cache misses
Prompt-cache reads cost a fraction of fresh input. A low cache-hit rate means you're re-sending the same context at full price every turn. Estimate: (your cache-hit % shortfall vs. a well-cached session) × input tokens × input rate × 0.9. Sessions that should be 80%+ cache and read 11% are the single most common silent overspend we see.
🔀 3. Silent model fallbacks
A session that quietly ran on two models — a fallback or downgrade you never chose — is both a cost signal (you got billed for the expensive one) and a quality signal (the cheap one may have done the work). Estimate: count sessions that mixed models, and for each ask whether the expensive model was actually needed. The save is the delta to running the right model on purpose.
⚠️ 4. Tool-failure loops
A tool that keeps erroring (a flaky MCP, a browser that 40%-fails) burns tokens on retries while you just see "thinking." Estimate: tool-failure rate × the tokens spent on the failed-and-retried turns. Fixing one chronically failing tool often pays for the whole month.
♻ 5. Compaction thrash
Every auto-compaction re-summarizes — and re-bills — the context window. A session that compacted many times is thrashing. Estimate: compactions × the tokens reclaimed each time, priced at input + cache-write. Often the fix is a smaller working context, not a bigger one.
🔄 6. Runaway loops
The most expensive failure mode: the same tool, same input, fired over and over, sometimes for hours, while the agent "thinks." Estimate: the per-loop token cost × loop length. A single caught runaway can dwarf every other line on this list — this is the category where catching one event pays for the tooling outright.
Add the six and you have your recoverable spend — the fraction of your agent bill that is waste rather than work. It varies enormously by setup, which is exactly why a generic "save X%" claim would be dishonest. The number that matters is yours, and it's sitting in your runs whether or not you're looking at it.
A worked example (illustrative — plug in your own numbers)
Suppose a small team spends $2,000/mo across a few coding agents. The figures below are hypothetical, purely to show the arithmetic — do not read them as a benchmark:
- Cache hit-rate lifted from ~40% to ~80% on long sessions → recover part of the input bill.
- Two chronically-failing tools fixed → stop paying for retried turns.
- One runaway loop caught per month before it ran overnight → a lumpy but real save.
- Routine edits moved off the most expensive model after seeing the model-mix chip.
In a plausible setup these compound to a double-digit percentage of the bill — but the only honest way to know your figure is to measure it. The point of the framework isn't the percentage; it's that each line is observable and addressable once it's on screen.
The other half: hours, not just dollars
The cost side is the easy half to quantify. The bigger productivity gain for most teams is time:
- Less log-digging. "What did the agent actually do, and where did it go wrong?" is a question you answer in a glance instead of by grepping JSONL across machines.
- Faster incident detection. A connector that goes deaf, an agent stuck in a loop, a tool failing — surfaced as it happens, not discovered in next month's invoice.
- Trust to run unattended. The real unlock of agents is letting them work while you don't watch. You can only do that safely if something is watching — and will flag the runaway before it blows the budget.
These are harder to put a single number on, but they're where the "productivity" in productivity gains actually lives: the same person can run more agents, more confidently, with less manual oversight.
How to measure your own number
You don't have to take any of this on faith. It's open source, read-only by default, and installs in about thirty seconds:
Then open the dashboard and look at your sessions. Each one carries its real cost and the six waste chips. Sort by cost, read the chips on your most expensive sessions, and you'll have your recoverable-spend number — computed from your runs, not ours — within an afternoon. Works with OpenClaw, NVIDIA NemoClaw, Claude Code, Codex, Cursor, Aider, Goose and more.
A note on numbers: every figure in the worked example above is illustrative and clearly labeled as such. We deliberately don't publish a headline "ClawMetry saves you N%" — the gains depend entirely on your agents, your plan, and what you fix once you can see it. The framework here is for estimating your number; the dashboard is where you'll find it.