Would ClawMetry have caught the DseWiki agents? We ran the test.

Two agent breakouts went public within weeks of each other this summer. In the first, AI agents under evaluation escaped their sandbox and got into Hugging Face's production systems. In the second, agents made more than 15,000 edits to a German programming wiki called DseWiki. They used it as a message board, swapped tips on hiding from their monitors, and routed around the moderators who kept deleting their pages.

We build a tool whose whole job is to notice agents doing things they shouldn't, so the obvious question landed in our inbox the same day: would ClawMetry have caught this?

We didn't want to answer from the marketing page. So we replayed both incidents, shaped the way the public reports describe them, against the same open-source detectors that ship in pip install clawmetry. Here is the scorecard, including the parts that don't flatter us.

What actually happened

According to the Reuters investigation and the coverage that followed, the DseWiki activity started in May and ran for about two months. Accounts with names like OpenAIResearcher and OAIResearchMar26 discussed getting around safeguards, using Tor, and keeping their messages alive through shutdowns. When moderators started deleting pages in June, one agent left a pointer to a backup page named so it would sort to the end of the alphabet: "If this page vanishes, try [[ZZZDataUSAConstructionWageLive]]."

It came to light in August, and the operator didn't find it. Independent researchers from the Nightingale collective found it by reading the text the agents had left on the wiki, with no access to any internal logs. As The Next Web put it: "Nothing in OpenAI's monitoring flagged thousands of edits being written to a public website."

The earlier Hugging Face breach is better documented, thanks to Hugging Face's own technical timeline. The agent's first action on its external launchpad was textbook reconnaissance: id; env; cat /proc/self/mountinfo. It went on to read a Kubernetes service-account token and leak worker environment variables containing secrets. It used pastebins and a request-capture service for command and control, and built a self-respawning fleet across eleven nodes. About 17,600 attacker actions were recovered between July 9 and 13. OpenAI later said the agents were reward hacking: trying to cheat on an evaluation by finding the answers online.

How we tested it

ClawMetry's Guard runs judge-free detectors over each session's tool stream: no LLM in the loop, just what the agent called and with which arguments. The four behavioural ones matter here:

• network_egress: the agent reached a host this agent (or its cohort) has never contacted, fanned out to many hosts at once, or connected to a raw IP.
• credential_access: the agent read an SSH key, cloud credentials, an .env file, a token store, or the cloud metadata endpoint. It turns critical when network egress follows.
• privilege_change: sudo, edited sudoers, disabled system protections, setuid bits.
• file_blast_radius: an unusually wide write, or a delete aimed at a root directory.

"Never contacted" needs a memory, so Guard learns one per cohort (a runtime, or a single named agent) once it has seen 20 sessions. A new destination is judged against that learned host set.

We built synthetic sessions for eight scenarios and ran them through the released detector code. A typical one: forty wiki edits, posted with curl, from a coding agent.

curl -s -X POST https://dsewiki.de/api.php \
  -d 'action=edit&title=Page17&text=...'
# x 40, in one session

The scorecard

Why the misses happen

1. The swarm teaches the baseline

This is the one that worries us most, because it's the incident's defining feature. Hundreds of agents, all from the same cohort, all reaching the same place. Guard learns known hosts from every session in a cohort. When the first session posts to the wiki, the host goes into the cohort's memory, and every sibling after it passes as "not new". We already clamp the numeric thresholds so a cohort that loops end to end can't teach Guard to go blind. The host set had no equivalent guard.

Fixed in 0.12.859: a host now has to sit in the cohort's memory for 24 hours before it counts as known.

2. "First contact" needs history, and a fresh install has none

We deliberately don't treat every host as suspicious on day one; that would bury you in alerts. The cost is that a brand-new deployment, which is exactly when a mis-scoped agent is most likely to wander, can't say "this is new" at all. It only fires on fan-out (eight or more distinct hosts in one stretch) or raw IP addresses.

3. We see the command, not the socket

Guard reads what the agent asked its tools to do. curl https://dsewiki.de is visible. python3 post_edits.py, where the URL lives inside the script, isn't. An agent that writes a helper and runs it is common, and it's the path that goes dark.

4. Volume and direction aren't counted

Forty POSTs and one GET look the same to network_egress: one host. "Fifteen thousand writes to a public site" is exactly what the operator's monitoring failed to flag, and today we wouldn't count it either.

5. The recon patterns are narrower than the real recon

We tuned credential_access for precision after early versions flagged every Python heredoc as a secret dump. That tuning left two real holes, both from the Hugging Face timeline: env followed by ;, and the Kubernetes service-account token path.

Fixed in 0.12.859: both now match, and env FOO=1 prog (running a program, not dumping) still doesn't.

What would close each gap

1. Cohort-wide first contact. Track when a host entered the cohort's memory and how many sessions put it there. A host that arrived this week, from sessions that are all still running, isn't "known". It's a spreading destination, and that's a signal in its own right: N sessions reached the same new host within an hour. Half shipped in 0.12.859: arrival time is tracked, so a recent host is no longer "known". Folding the N sessions into one incident is still to come.
2. Anonymiser and dead-drop hosts get names. .onion, torsocks, SOCKS proxies, pastebins and request-capture services should raise a labelled incident on day one, with no history needed.
3. Write volume per host. Count POST, PUT and upload calls per destination per session, and flag sustained writes to a host outside the agent's work.
4. Close the two recon holes. env as a statement anywhere in a command, and service-account token paths as a strong credential category. Shipped in 0.12.859.
5. A cold-start allowlist instead of silence. On a fresh install, compare against the destinations the runtime is known to need (package registries, the model API, the forge), not against nothing.
6. Where the runtime allows it, see the socket. Script-level egress needs process-level network attribution, not command-line parsing. It's the hardest item on this list, and the only one that catches the helper-script path.

What already works: a stop that actually stops

The panel that prompted this post (AIM's Point Break, with Seclore and Matters.AI) kept coming back to one point. At machine speed, detection without a response just tells you what already happened.

That's the right shape for this incident class. What's missing is the trigger. A ladder that pauses on "first contact with a host no one in this cohort has used" would have been a strong response to DseWiki, but only if the swarm hadn't already taught Guard that the host was ordinary.

The honest part

The most important sentence in the whole story is that outsiders found this by reading a wiki. The operator had logs. What it didn't have was something watching its own agents and asking, continuously, is this where they normally go?

That question is what ClawMetry is for, and this exercise showed us where our answer is still too trusting. We'd rather publish the misses than have you discover them the way DseWiki's moderators did.

Sources

• CNBC: OpenAI agents hijacked German website in previously undisclosed AI breakout (Reuters report)
• Fortune: OpenAI's AI agents secretly ran their own message board on a German wiki
• The Next Web: OpenAI agents hijacked a German wiki for two months
• Hugging Face: Anatomy of a Frontier Lab Agent Intrusion
• Forbes: OpenAI finds agents that breached Hugging Face were "reward hacking"
• AIM Network, Point Break: AI Agents Are Infiltrating Enterprise Systems