Skip to Content
Getting started

Getting started

ablate measures whether your CLAUDE.md actually does anything — and tells you which parts of it you can delete.

Run a coding agent N times under config A and config B, check outcomes with deterministic assertions, and report the delta with a confidence interval. Then attribute that delta to individual sections. Not “is config A better than config B”, but which of these lines can I delete.

No LLM judge, anywhere. No hosted service, no accounts, no telemetry.

Install

uvx --from ablate-cli ablate demo

The distribution is ablate-cli — ablate on PyPI belongs to an unrelated deep-learning reporting tool. The command you type is still ablate.

From source:

uv venv && uv pip install -e ".[dev]" && source .venv/bin/activate
ablate --help

The five-minute path

Start with the two commands that cost nothing:

ablate demo                  # replay a bundled run — no API key, no network
ablate cost CLAUDE.md        # what your config costs on every turn, exactly

Then measure your own repo:

ablate init                  # scaffold .ablate/tasks/
# fill in the TODOs in .ablate/tasks/*.yaml
ablate try                   # smallest real run: 18 trials on Haiku, <= $2
ablate compare CLAUDE.md /dev/null --trials 5

Then attribute the result to individual sections:

ablate sections CLAUDE.md              # which sections earn their keep
ablate apply --drop s11,s06 CLAUDE.md --write

compare, try and sections need the claude CLI on PATH and an authenticated session or ANTHROPIC_API_KEY set. Each prints the run count, the estimated cost, and the effect size the design can actually resolve before executing anything, and stops at --max-spend.

Full detail on every command: CLI reference.

How it decides

Every comparison resolves to exactly one of four verdicts, from the confidence interval and an equivalence margin m (default ±5pp; --margin to change).

VerdictWhenMeans
🔴 REGRESSIONinterval sits below −mA measurably worse than B
🟢 IMPROVEMENTinterval sits above +mA measurably better than B
🔵 EQUIVALENTinterval fits inside ±mBounded. Safe to prefer the cheaper config.
🟡 INCONCLUSIVEanything elseNot enough data to say — here’s what it would cost

EQUIVALENT and INCONCLUSIVE are opposite findings. “We looked hard and there is nothing there” licenses a deletion; “we didn’t look hard enough” licenses nothing. They never share a colour, a sentence, or a code path — and every INCONCLUSIVE verdict ends with the tasks, trials and runs that would resolve it.

What it costs

CommandCostTimeWhat
ablate demo$03 sBundled run. No API key, no network.
ablate cost$0instantPer-section token cost. Exact arithmetic.
ablate try≤ $2~6 min3 tasks × 3 trials on Haiku. Smoke test, not a measurement.
ablate compare~$28~25 min8 tasks × 5 trials × 2 arms on Sonnet.
ablate sections~$110~90 minScreen on Haiku + confirm on Sonnet, K=12.

Every spending command prints its estimate and its resolvable effect size before it starts, and honours --max-spend (default $25). In CI the ceiling is mandatory and breaching it fails the job.

Writing a task

A task is a prompt plus deterministic checks, pinned to a commit. ablate init scaffolds three to start from. Full field reference: Task & config schema.

id: convention-cost-json
description: Add a --format json option to `ablate cost`.
 
setup: uv venv -q && uv pip install -q -e ".[dev]"
prompt: |
  `ablate cost` prints a human-readable table. Add a `--format text|json`
  option. Follow the conventions already used by the other commands.
 
assert:
  - id: tests-pass
    type: exit_code
    run: .venv/bin/python -m pytest tests/test_cli.py -q
 
  - id: test-added
    type: files_touched
    must_include: ["tests/**"]
 
  - id: no-bare-except
    type: absent
    pattern: 'except\s*:'
    in: diff
 
tags: [convention, smoke]

What it can’t tell you

The honest limits, up front rather than after you have spent $28.

  • Pass rate is a guardrail, not a headline. Published pass-rate effects of config changes are 3–4 percentage points. A default 8-task × 5-trial run resolves about 32 points. It will usually return INCONCLUSIVE on pass rate, and says so instead of pretending otherwise.
  • Token cost is where the signal is. The same run resolves a ~16% change in tokens per turn, and measured cost effects are 16–28%.
  • Below 8 tasks there is no confidence interval at all. ablate init scaffolds three and ablate try runs three — enough to prove the plumbing, not to measure anything.
  • Δ is an average over your suite, for your agent at your model version. It is not a general claim about the config.
  • Screening is a ranking, not a measurement. Screening intervals cover ~88% rather than the nominal 95%. Read the order of the screening table, not its numbers — which is why the confirmatory stage is not optional.
  • Attribution is section-level with line ranges, not per-line. --granularity bullet makes it literally per-item and roughly triples the cost.

In CI

- uses: ablate-dev/ablate@v1
  with:
    config: CLAUDE.md
    baseline: origin/${{ github.base_ref }}
    max-spend: 15          # required — no default, on purpose
    fail-on: never         # regression | inconclusive | never
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Posts a sticky PR comment whose footer states the MDE for the run — how large a regression would have had to be before that green check could have caught it. Start with fail-on: never; the first thing most teams learn is that their suite is underpowered for pass rate, which is worth knowing before it gates a merge.

Where to go next

Last updated on
Overview