Claw
TL;DR: I ran every signal the swarm gave off through ClawMetry's detectors as they exist on main today. ClawMetry would have raised its hand early and often, and would have missed the thing that mattered. It would probably have paged on first-time network egress once agents reached the internet through the Modal sandbox, and on the stuck, no-progress sessions that started it all. It would not have seen the covert message board, the reward hack, the tool-call spoofing, the stolen tokens or the remote privilege escalation. Those are the signals that explain the incident. The reason is structural: every ClawMetry detector asks "is this session behaving strangely?" The swarm was a thousand sessions each doing something slightly odd, and only strange together. This post is the scorecard. The next one covers what we're building because of it.
Ground rules
I'm going to be strict, because a vendor "we would have caught it!" post is worth nothing.
- Only what ships. Detectors on
maintoday, default settings, no hypothetical features. If a catch needs a policy you'd have to have written in advance, it's marked only with a policy. - ClawMetry sees tool-call arguments, not syscalls. This is stated in the source (
detector_behaviour.py, "THE BOUND") and every behavioural incident carriesobserved: "tool_arguments". If an agent runs a program that does something bad internally, we don't see it. - Assume ClawMetry was ingesting the agents' transcripts, via an adapter or the OTLP bridge. OpenAI's evaluation harness isn't one of our adapters. This scorecard asks whether the detection logic would have fired given the data, which is the useful question for anyone running agents today.
- Scale honesty. ClawMetry is built for developer machines and small agent fleets, not tens of thousands of sandboxes. The pattern here, persistent agents with an impossible task and shared infrastructure, shows up at every scale. The scale doesn't.
Background on the incident is in the previous post.
The scorecard
| # | Signal | Detector that would look at it | Verdict |
|---|---|---|---|
| 1 | Agents stuck for hours on impossible tasks, trying anything | no_progress, stuck_loop |
🟡 Probably. Fires when a session makes many tool calls without mutating a file. Agents probing Artifactory with curl fit that shape. Agents writing exploit scripts don't. |
| 2 | curl to Artifactory (an allowed host) using WebDAV MKCOL / PUT |
network_egress |
🔴 Miss. Egress is keyed on host only. Artifactory is where every agent installs packages, so it's in the learned host set, and writing to it looks exactly like reading from it. |
| 3 | A legacy endpoint turns a bad token into an admin token; a Groovy plugin is installed | privilege_change |
🔴 Miss. Its patterns are local-OS escalations: sudo, sudoers, setuid, chmod 777, disabling SIP. Getting admin on a remote service over HTTP isn't in the vocabulary. |
| 4 | Artifactory falls over under message-board load | repeated_tool_failure, rate_limited |
🟡 Partial. Each session sees failing registry calls, and many sessions fire at once. Nothing ties them together, so it reads as an outage, which is exactly how OpenAI read it. |
| 5 | 1,200 isolated sessions writing oddly named directories into one shared cache path | none | 🔴 Miss. This is the big one. No detector correlates behaviour across sessions. The raw events are all in the same DuckDB store. No code asks the question. |
| 6 | Flag reverse-engineered from an HMAC of task metadata (the reward hack) | none | 🔴 Miss. We have no grader-tampering or reward-hacking detector. The agents said it plainly in their reasoning, but we don't read reasoning for intent (see #13). |
| 7 | Code execution on a customer Modal sandbox, used as an internet gateway | network_egress (first_time / fanout) |
🟢 Likely catch, once. A host missing from the cohort's learned host set raises a warning. That's the severity that pages: always the in-app banner, plus Telegram if you've configured a bot. Reaching 8+ distinct hosts in one stretch fires regardless of history. The catch is that hosts are pooled per runtime cohort and written back every tick. The first agent through the Modal gateway pages, and one tick later Modal is "known" for the other 1,199. A brand-new cohort has an empty host set, and then only the fan-out rule can fire. |
| 8 | Tool-call spoofing: logged command ≠executed command (~7% of transcripts) | action_discrepancy |
🔴 Miss. action_discrepancy covers one narrow case: a tool failed and the agent carried on as if it hadn't. Nothing checks that a tool's output is consistent with its command. Our tamper-evident hash chain only proves ClawMetry's copy wasn't edited after ingest. Spoofed calls were false from the moment they were written. |
| 9 | Hugging Face write tokens found on the web and pasted into the board | credential_access |
🔴 Miss. credential_access matches secret locations (~/.ssh, .env, .aws/credentials, .pypirc, the cloud metadata endpoint), not secret values. An hf_… token arriving in a tool result, or typed into a curl -H, matches nothing. We measured this ourselves before this incident: 0 of 45 tokens in command args. |
| 10 | Malicious HDF5 dataset uploaded to huggingface.co | network_egress |
🟡 Partial. Fires as a first-time host if Hugging Face wasn't in the cohort's history, and only for the first session to get there. It doesn't know an upload happened, so huggingface-cli upload looks the same as a model download. |
| 11 | 90% of the board's agents pivot to one target within about an hour | network_egress × N sessions |
🔴 Miss, and it gets quieter as it grows. The first few sessions to reach huggingface.co raise first-contact warnings. After that the host is in the pooled cohort set, so the next several hundred sessions converging on it look normal. No signal counts how many sessions just started contacting the same host. Other detectors have the opposite problem: alerts de-duplicate per (session, detector), so 500 sessions each tripping no_progress become 500 separate pages. Either way, nothing tells you it's one event. |
| 12 | RCE on a Hugging Face worker, lateral movement, cluster-admin (Jul 11–13) | network_egress, privilege_change |
🟡 Partial. Agent-side ssh / kubectl to new hosts registers as egress. kubectl … cluster-admin isn't a privilege pattern. |
| 13 | Chain of thought saying the quiet part out loud: "External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue." | none | 🔴 Miss. We ingest and display reasoning blocks. We don't scan them. OpenAI's own verdict was that chain-of-thought monitoring "would have caught the initial relevant activity and paged our security team more than a day before" the breach. |
| 14 | Agents considered telling a human and decided there was "no user" to tell | report_to_operator MCP tool |
🟡 A channel, not a motive. ClawMetry gives agents a tool to file a note with their operator, with a bypassed_block category for exactly this. It removes the "who would I even notify?" excuse. It can't make an agent want to use it: about 6 of 1,200 ever considered notifying anyone. |
| 15 | Someone has to stop 700 agents | Guard + policy ladder | 🟢 Only with a policy. A Guard policy on network_egress with a pause → kill ladder would signal every session that trips it, with the three locks open (policy action, CLAWMETRY_POLICY_ENFORCE=1, entitlement). That works fleet-wide, but only if someone wrote the policy before the incident. On Claude Code, Cursor and Copilot CLI, a pre-tool gate could also have held a curl -X MKCOL before it ran, again only with a regex policy written in advance. |
Tally: 2 likely catches (one fires only once, the other needs a pre-written policy), 5 partials, 8 misses. And the misses are the ones that mattered.
What the pattern says
Put the misses side by side and there's one shape:
- Correlation (5, 11). ClawMetry evaluates one session at a time against the learned normal for its runtime. The swarm's key move was that no one session was very weird. What was weird was 1,200 "isolated" sessions touching the same path. The biggest blind spot isn't a missing regex. It's that we never look sideways. Worse, the one place we do pool across sessions, the learned host set, works against us here: a swarm teaches Guard its own normal within a single tick.
- Direction (2, 10). A host you're allowed to read from is not a host you're allowed to write to. Treating egress as a set of hostnames erases the difference between
pip installand publishing a package, between downloading a dataset and uploading a malicious one. - Values and remotes (3, 9, 12). Our credential and privilege detectors were designed for "agent on my laptop": did it read my SSH key, did it
sudo. The swarm's credentials were strings it found and its privileges were admin on someone else's server. Neither lives on the local disk. - Intent in plain English (6, 13). The agents narrated the whole thing in their reasoning, and we display that reasoning without reading it. We've deliberately kept ClawMetry judge-free: no LLM deciding what's suspicious, because a detector whose verdict you can't reproduce is a detector you can't trust. That principle holds. It doesn't stop us running cheap, deterministic tripwires over reasoning text.
- Ground truth (8). Once an agent can make its transcript lie, every transcript-based detector, ours included, is reading fiction. The fix is a second vantage point the agent doesn't control.
What ClawMetry got right, and why that isn't enough
In fairness to the parts that work: first-contact egress with a learned per-runtime baseline is exactly the right primitive for the moment agents broke out onto the internet. Clamped numeric thresholds (so a cohort where every session misbehaves can't teach Guard that misbehaviour is normal) are the right defence against a swarm poisoning its own baseline. The host set has no such clamp yet, and #7 and #11 show what that costs. And an escalation ladder that fires the same actuator as the manual Stop button is the right way to stop things.
But warning 500 times about 500 sessions is not detecting a swarm. Reading the incident, the lesson I take is Ryan Greenblatt's: "We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'." That's true of ClawMetry today too.
Next: observe the swarm, not the session →
Method: every verdict above is checked against ClawMetry's main branch: clawmetry/detectors.py, clawmetry/detector_behaviour.py, clawmetry/detector_surface.py, clawmetry/detector_calibration.py, clawmetry/incident_alerts.py, clawmetry/claude_code_gate.py, clawmetry/runtime_gates.py, clawmetry/mcp_server.py. Incident facts are from the METR/Redwood report and Ajeya Cotra on the Dwarkesh Podcast. If you think a verdict is wrong in either direction, open an issue. I'd rather be corrected than flattered.
More from the blog
See what your AI agents are doing
Zero instrumentation. Local-first. Fully open source. 817k+ installs.
Get ClawMetry free