---
title: eval
description: clawmetry eval — run golden eval suites against your agents, watch mode for the dev loop, JSON output for CI, and regression replay of last week's failures.
keywords: agent eval suite, LLM regression testing, agent CI evaluation, golden tests for agents
eyebrow: CLI reference
---

# `clawmetry eval`

Run a golden eval suite. Suites are YAML files in `~/.clawmetry/evals/`.

```bash
clawmetry eval --list
clawmetry eval --suite customer_support
```

## Flags

| Flag | Effect |
|---|---|
| `--suite NAME_OR_PATH` | Suite name, or an absolute path to a YAML file. Required unless `--list`. |
| `--list` | List available suites and exit |
| `--watch` | Re-run on every change to the suite file |
| `--json` | Machine-readable JSON instead of the table |
| `--no-persist` | Do not write results to DuckDB — useful for dry runs |
| `--regression` | Replay last week's failed sessions against the current configuration |
| `--window DURATION` | Lookback for `--regression` (default `7d`) |
| `--limit N` | Hard ceiling on replays per `--regression` run (default: `CLAWMETRY_EVALS_REGRESSION_MAX` or `10`) |

## Exit codes

`0` on all-pass, `1` on any failure. That is the contract CI depends on —
nothing else about the output is load-bearing for a pipeline.

```bash title="In CI"
clawmetry eval --suite smoke --json > eval.json || {
  jq -r '.results[] | select(.passed | not) | "FAIL: \(.name) — \(.reason)"' eval.json
  exit 1
}
```

## The dev loop

```bash
clawmetry eval --suite my_suite --watch
```

Re-runs on every save of the suite file. `--json` is honoured on each iteration,
so you can pipe a watch loop into `jq` while you iterate on the suite.

Pair it with `--no-persist` while you are still shaping the suite, so half-baked
runs do not land in your results history.

## Regression replay

```bash
clawmetry eval --regression --window 14d --limit 25
```

Takes sessions that failed in the window and replays them against your current
configuration — the answer to "did that change actually fix it, or did I just
stop seeing it?"

This costs real money, because it makes real model calls. That is why the limit
is a hard ceiling rather than a suggestion, and why the whole path is gated:

| Variable | Effect |
|---|---|
| `CLAWMETRY_EVALS_ENABLED` | Master switch for the eval subsystem |
| `CLAWMETRY_EVALS_REGRESSION_ENABLED` | Separate switch for the replay path |
| `CLAWMETRY_EVALS_REGRESSION_MAX` | Default replay ceiling |
| `CLAWMETRY_EVALS_AGENT_CMD` | The command used to run a replay |

`--regression` and `--suite` are mutually exclusive — they produce different
table shapes, and mixing them in one invocation only confuses CI logs.

## Results

Runs persist to DuckDB unless `--no-persist`, and are visible in the dashboard
alongside session data — so an eval failure is one click from the transcript
that produced it.

```bash
curl -s localhost:8900/api/evals/recent | jq
curl -s localhost:8900/api/evals/summary | jq
curl -s localhost:8900/api/evals/regression-summary | jq
```

## Tier

`eval_suite` is a Pro feature.