CClawMetryDocs

CLI reference

clawmetry eval#

Run a golden eval suite. Suites are YAML files in ~/.clawmetry/evals/.

bash
clawmetry eval --list
clawmetry eval --suite customer_support

Flags#

FlagEffect
--suite NAME_OR_PATHSuite name, or an absolute path to a YAML file. Required unless --list.
--listList available suites and exit
--watchRe-run on every change to the suite file
--jsonMachine-readable JSON instead of the table
--no-persistDo not write results to DuckDB — useful for dry runs
--regressionReplay last week's failed sessions against the current configuration
--window DURATIONLookback for --regression (default 7d)
--limit NHard ceiling on replays per --regression run (default: CLAWMETRY_EVALS_REGRESSION_MAX or 10)

Exit codes#

0 on all-pass, 1 on any failure. That is the contract CI depends on — nothing else about the output is load-bearing for a pipeline.

In CI
clawmetry eval --suite smoke --json > eval.json || {
  jq -r '.results[] | select(.passed | not) | "FAIL: \(.name) — \(.reason)"' eval.json
  exit 1
}

The dev loop#

bash
clawmetry eval --suite my_suite --watch

Re-runs on every save of the suite file. --json is honoured on each iteration, so you can pipe a watch loop into jq while you iterate on the suite.

Pair it with --no-persist while you are still shaping the suite, so half-baked runs do not land in your results history.

Regression replay#

bash
clawmetry eval --regression --window 14d --limit 25

Takes sessions that failed in the window and replays them against your current configuration — the answer to "did that change actually fix it, or did I just stop seeing it?"

This costs real money, because it makes real model calls. That is why the limit is a hard ceiling rather than a suggestion, and why the whole path is gated:

VariableEffect
CLAWMETRY_EVALS_ENABLEDMaster switch for the eval subsystem
CLAWMETRY_EVALS_REGRESSION_ENABLEDSeparate switch for the replay path
CLAWMETRY_EVALS_REGRESSION_MAXDefault replay ceiling
CLAWMETRY_EVALS_AGENT_CMDThe command used to run a replay

--regression and --suite are mutually exclusive — they produce different table shapes, and mixing them in one invocation only confuses CI logs.

Results#

Runs persist to DuckDB unless --no-persist, and are visible in the dashboard alongside session data — so an eval failure is one click from the transcript that produced it.

bash
curl -s localhost:8900/api/evals/recent | jq
curl -s localhost:8900/api/evals/summary | jq
curl -s localhost:8900/api/evals/regression-summary | jq

Tier#

eval_suite is a Pro feature.

Edit this page View as Markdown Report an issue
Cookie preferences