Tool Calls That Returned 200 and Still Failed

There is a class of AI agent failure that is invisible to every monitoring tool designed for deterministic services. Your uptime check passes. Your API logs are clean. Your error rate is zero. The HTTP status is 200. The JSON parsed without exception. The tool call, as far as every layer of your stack is concerned, succeeded.

And the agent built its entire output on bad data, then committed the result.

I want to be precise about what distinguishes this from the failures I wrote about in previous posts. A rate-limit loop is visible as cost. A hanging process is visible as absence. A stale cached response is at least a semantics problem you can reason about. What I am describing here is different: the tool call reports success at every level of abstraction, and the failure lives entirely in the content of the response — content that only becomes meaningful in the context of what the agent does next.

Here are three patterns that keep showing up. Each one is drawn from real developer reports in the past 30 days. I have paraphrased and combined similar accounts to protect privacy.

Incident 1: The Null That Passed as a Value

This one is subtle because it requires understanding the difference between a transport error and a semantic contract violation. The API returned HTTP 200. The JSON was well-formed. No field was missing — the field was present, and its value was null. From every angle that traditional monitoring watches, the call succeeded.

The agent was running a purchasing workflow: query current prices, calculate margin, submit order if margin exceeds threshold. The pricing API's null for an out-of-stock item was intended by the API designers as a signal to check back later. The agent, reasoning from its tool response, interpreted null as a valid numeric input and proceeded through its calculation chain. In Python and most interpreted runtimes, arithmetic on None raises a TypeError. But the agent was not doing arithmetic in Python — it was constructing a reasoning chain in natural language, where null became the string “null” and then, in the model's arithmetic reasoning, became 0.

Source: Pattern synthesized from Openlayer's 2026 agent failure modes analysis and developer reports; the null-as-zero reasoning failure is a well-documented LLM arithmetic edge case.

The purchase order went out at a margin the business would never approve. No alarm fired. Nothing in the API logs looked wrong. The agent's final output was a fully formatted order with references to the correct SKU, the correct vendor, and a calculated margin that was, quietly, fictional.

What ClawMetry surfaces

The honest answer is: if you do not have the raw tool response stored, this failure is nearly impossible to diagnose after the fact. Your API logs show a 200. Your agent logs show a successful completion. The order submission endpoint accepted the request. There is no error to grep for. You need the verbatim model input at every tool call step, in sequence, and you need it before something downstream acts on it.

Incident 2: The Schema That Drifted Without a Version Bump

This pattern is documented in enough production postmortems now that I expect it to have a name soon. A third-party API ships a “non-breaking” change: they add a wrapper envelope around an existing payload. Every HTTP status is 200. Every JSON body is valid. Every existing field still exists, one level deeper. Clients that access fields by name through a typed SDK pick up the breakage immediately — their deserialization throws. An AI agent does not throw. It reads the JSON, sees a dict where it expected a dict, and proceeds. The values it extracts are the wrapper's keys, not the payload's keys.

Source: DEV.to, “Building Production-Ready AI Agents in 2026,” which specifically names schema drift as one of the most common silent failures in production agent deployments.

The eleven-day window before discovery is what I want to draw attention to. For the first day or two, the reports the agent generated looked plausible — they had the right structure and referenced real field names from the wrapper envelope. Only when a human tried to act on a specific figure did the problem surface: the number was not in the source data at all. Eleven days of agent runs, none of which produced a single error.

The transcript viewer shows what the model actually received at every tool call step. Schema drift becomes visible as a mismatch between the response shape and the agent's downstream assertions.

What ClawMetry surfaces

The underlying lesson is that “non-breaking” is a concept defined relative to typed clients, not AI agents. AI agents read JSON in natural language context and will make plausible inferences about a field named result that holds a dict. Those inferences will be wrong in a specific, consistent way — which is actually helpful for diagnosis if you have the raw responses. Without them, you're trying to deduce the schema change from the agent's output, which is an underdetermined problem.

Incident 3: The Tool Definition That Fell Out of Context

This is the one that unsettles me most, because the failure mode is the absence of a tool call, not a bad one. The agent did its job. It reasoned about the data, it wrote the report, it said “submitted.” It just never called the tool. There was no error to catch because there was no call to fail.

Context window saturation in long sessions is well-documented. When the context fills, compaction kicks in: earlier turns are summarized or compressed. Tool definitions loaded at the start of the session may be summarized away. The model does not know what it has lost — it reasons from what it has. If submit_report is no longer in scope, the model writes the report as prose and considers the step done, because writing a report is what a report-writing agent does.

Source: DEV Community, “Why AI Agents Fail in Production,” 2026; context-window tool loss is named as one of the three most common sources of undetected agent failure.

The receiving system never got a call. The agent's output log showed successful report generation. The gap was invisible until someone checked the downstream database and found reports missing from a three-hour window. The agent had been producing beautifully formatted reports and printing them into the void.

What ClawMetry surfaces

Context compaction is a feature, not a bug. Long-horizon agents need it. The problem is that it creates a class of tool-availability failure that has no error signal at the tool layer — the tool was never called, so nothing failed. The only place this failure is detectable is in the negative space between what the agent said it did and what the tool call log shows it actually did. If you are not keeping that log, you have no way to close the loop.

The pattern across all three

I want to be precise about what these three incidents have in common, because it matters for how you instrument against them.

In all three cases, the agent's external behavior looked correct:

• Incident 1: The agent completed the workflow and submitted an output. The purchase order was formatted correctly.
• Incident 2: The agent completed eleven days of reports, each one well-structured and plausible-looking.
• Incident 3: The agent completed the analysis and stated it had submitted the result. No error occurred.

Traditional monitoring — process health, API status codes, error rates, latency — was clean in all three cases. The failure lived entirely in the content of what the agent received or produced, which is the one layer that traditional monitoring does not inspect.

The implication is uncomfortable but important: for AI agents, a green health check is a weaker signal than it is for deterministic services. A web server that returns 200 either served the right content or it didn't; you can usually distinguish them at the HTTP layer. An agent that completes a workflow either did the right thing or it didn't, and the distinction often requires reading the verbatim tool inputs and outputs in sequence.

What we ship for each of these

I want to be concrete about what ClawMetry actually provides, because I don't want this to read as marketing-by-implication.

For null propagation and semantic contract violations

The transcript viewer at /api/transcript/<id> stores every tool call request and response verbatim, in sequence, as part of the session record in the local DuckDB store. This requires no instrumentation of the agent code; it is derived from the session transcript files the agent runtime already writes. For Claude Code sessions, that means ~/.claude/projects/<slug>/*.jsonl. For OpenClaw, the equivalent JSONL files. No SDK, no code changes.

For schema drift

The historical DuckDB view makes this queryable. If you want to know when a tool's response structure changed, you can scan the stored transcripts for that tool across sessions and look for when new top-level keys appeared. The behaviour signals layer runs this kind of structural comparison automatically against the cohort baseline.

For context-window tool loss

The brain stream at /api/brain-stream records reasoning events in real time. The tool call timeline records what was actually called. Guard policies let you assert invariants over both: “this session type must call submit_report at least once per hour.” A breach surfaces in the Guard tab before the session completes.

All three of these capabilities are part of the open-source core. The real-time alerts and Slack/PagerDuty integrations that let you act on them without opening a dashboard are Cloud Pro features. The data is always yours, local-first, on your own machine.

The honest limits

There are things ClawMetry does not do. We store the verbatim tool response; we do not validate it against a schema contract you have not declared. For null propagation, we surface the null — you have to build the policy that says “null in this field is a breach.” For schema drift, we surface the structural change — you have to know what the correct shape was.

We are an observability layer, not a contract enforcement engine. The schema contract is yours to declare. What we give you is the raw material to detect when that contract was violated, retroactively and in real time, without requiring you to instrument the agent code or change a single line of your workflow.

The three incidents above were discovered days or weeks after they began. With ClawMetry, any one of them could have been discovered at the first session where the failure occurred. That is not because we built specific detectors for null values or schema drift. It is because we store the verbatim tool call content and make it queryable. The failure is visible the moment you look at the data. The problem, without purpose-built agent observability, is that no one was looking at the right layer.