Task & config schema

A task lives at .ablate/tasks/<id>.yaml.

Task fields

Field	Type	Default	What
`id`	string	required	Unique within the directory. Used in `--tasks-only`, `--drop`, and everywhere the CLI prints a task name.
`description`	string	`""`	Human-readable summary. Not shown to the agent.
`base_commit`	string	current `HEAD`	The commit the task’s worktree is checked out at.
`setup`	string	none	Shell command run once per trial to prepare the environment (e.g. `uv venv -q && uv pip install -q -e ".[dev]"`).
`timeout_s`	int	`600`	Wall-clock budget for the agent on this task.
`prompt`	string	required	What the agent is told to do.
`config_path`	string	`"CLAUDE.md"`	Where the config variant is written in the trial’s worktree.
`hide`	list of globs	`[]`	Files the agent must not see (e.g. the answer, if the task was mined from a PR).
`assert`	list of assertions	required, ≥1	The deterministic checks. See below.
`weight`	float	`1.0`	Relative weight in the aggregate metric.
`tags`	list of strings	`[]`	Free-form labels for `--tasks-tag`.

Assertion types

Six types. All deterministic, all runnable offline, all checkable by hand — ablate never uses an LLM judge.

Type	Fields	Checks
`exit_code`	`run`, `expect` (default `0`), `timeout_s`	A shell command’s exit status. The workhorse — usually your test suite.
`files_touched`	`must_include`, `must_not_include` (globs)	The set of files the diff touched, against `base_commit`. Catches scope creep and unrequested edits.
`present` / `absent`	`pattern` (regex), `paths` (globs), `in` (`diff` or `files`, default `diff`)	Whether a regex matches — a single search over the concatenated diff (or file contents), not a per-file check.
`lint`	`run`, `expect`	Structurally identical to `exit_code`; kept separate so reports can distinguish correctness from style.
`script`	`run`, `expect`, `capture_json`	Escape hatch — any executable, run with `ABLATE_TRIAL_DIR`, `ABLATE_BASE_COMMIT`, `ABLATE_DIFF_FILE` in the environment. With `capture_json: true`, stdout is parsed as JSON and merged into the trial record.

Every assertion also takes id (required) and required (default true — set false for a guardrail check you want reported but not counted toward pass/fail).

Full example


id: convention-cost-json
description: Add a --format json option to `ablate cost`.
 
base_commit: 3f9a1c2
setup: uv venv -q && uv pip install -q -e ".[dev]"
timeout_s: 600
 
prompt: |
  `ablate cost` prints a human-readable table. Add a `--format text|json`
  option. Follow the conventions already used by the other commands.
 
hide:
  - tests/test_cost_json.py    # the answer, if mined from a real PR
 
assert:
  - id: tests-pass
    type: exit_code
    run: .venv/bin/python -m pytest tests/test_cli.py -q
    expect: 0
 
  - id: test-added
    type: files_touched
    must_include: ["tests/**"]
 
  - id: no-vendored-deps
    type: files_touched
    must_not_include: ["vendor/**", "**/node_modules/**"]
 
  - id: no-bare-except
    type: absent
    pattern: 'except\s*:'
    in: diff
 
  - id: lint
    type: lint
    run: .venv/bin/ruff check .
    expect: 0
 
weight: 1.0
tags: [convention, smoke]

passed for a trial is the conjunction of every required: true assertion.

Designing tasks that actually discriminate

A task every trial passes, or every trial fails, contributes zero information and costs full price — the report flags these as degenerate. The archetype that discriminates best is convention-adherence: “add a new Y,” checked with present/absent against the repo’s own idioms. It’s the one most sensitive to what’s actually in your config, which is exactly what you’re trying to measure.

ablate init scaffolds three starting tasks covering bug-fix, convention-adherence, and restraint. Below 8 tasks, no confidence interval can be produced at all — that floor is fixed, not a setting.