ablate compare
Run TASKS under CONFIG_A and CONFIG_B, N trials each, and report the
delta with a confidence interval. ~$28 default, ~25 min.
ablate compare CLAUDE.md /dev/null --trials 5
ablate compare CLAUDE.md CLAUDE.md.new --max-spend 15CONFIG_A and CONFIG_B are both plain file paths — there is no git-ref
syntax. CONFIG_B defaults to /dev/null (no config) — “is this file doing
anything at all.” A path that doesn’t exist is silently treated as
/dev/null too, so a typo in CONFIG_B reads as “compare against nothing”
rather than an error.
To compare against a historical version of the config, materialize it to a file first:
git show origin/main:CLAUDE.md > /tmp/claude-base.md
ablate compare CLAUDE.md /tmp/claude-base.md(This is exactly what the GitHub Action does under
the hood for baseline: origin/main.)
Arguments & flags
| Argument/flag | Default | What |
|---|---|---|
CONFIG_A | required | Config under test. |
CONFIG_B | /dev/null | Baseline to compare against. |
--trials N | 5 | Replicates per (task, variant). |
--tasks DIR | .ablate/tasks | Directory of task YAML files. |
--tasks-only IDS | none | Run only these task ids or globs. |
--tasks-tag TAGS | none | Run only tasks carrying any of these tags. |
--agent NAME | claude-code | claude-code or codex. |
--model ID | Sonnet 5 | Model id to run against. |
--cheap | off | Run on Haiku 4.5 (~3.5× cheaper). |
--concurrency N | 4 | Parallel trials. |
--max-turns N | 40 | Turn budget per trial. |
--max-spend USD | 25 | Hard ceiling, checked against real reported costs mid-run. |
--margin PCT | 0.05 | Equivalence margin, applies to every metric. |
--alpha A | 0.05 | Significance level. |
--seed N | none | Shuffle and bootstrap seed. Not agent sampling. |
--yes | off | Skip the confirm prompt. Never bypasses --max-spend. |
--keep-worktrees | off | Don’t clean up the trial worktrees afterward. |
--fresh | off | Ignore cached trials and re-run everything. |
--format FMT | text | Also write md, json, junit, or pr-comment into the run directory. |
--fail-on WHEN | never | regression, inconclusive, or never — exit non-zero on this verdict. |
What it does
Requires the claude CLI on PATH, authenticated or with
ANTHROPIC_API_KEY set — it calls the agent once per trial inside an
isolated git worktree, spending real money against your own account.
Before executing anything, it prints the run count, the estimated cost, and the effect size the design can actually resolve — so you can decline a run that can’t answer your question before paying for it.
Trials are memoized by (task, config hash, agent, model, replicate), so
editing one section of a config and re-running only pays for the arm that
changed. --fresh bypasses that cache.
The report resolves to one of the four verdicts per metric — pass rate, tokens per turn, turns to completion — with a confidence interval or an explicit statement of why one couldn’t be produced.