Add a task
A task is a two-file package. Copy tasks/demo/hello/ (Linux) or
hello_win/ (Windows) and edit. They cover the whole
input / reference / output surface in ~250 lines. The
Task spec & data staging page
covers the concepts.
tasks/<domain>/<task>/
├── task_card.json # metadata + the VM it needs
└── main.py # load() / start() / evaluate()task_card.json
| Field | Value |
|---|---|
taskId | <domain>/<task>, must match the folder path. |
category | The top-level domain. |
vm.snapshot | Which image to boot: cpu-free-ubuntu (Linux), cpu-free (Windows), gpu-free, cpu-license, gpu-license. |
vm.{vcpus,memory_gb,disk_gb} | Sizing hints, resolved against the snapshot's capacity pool. |
vm.timeout_s | Per-variant wall budget (provision + setup + agent + eval). Allow ~2× the agent's time. |
main.py: three hooks
| Hook | Runs | Does |
|---|---|---|
load() | at discovery | Returns one cb.Task per variant, each carries the prompt and the metadata the other hooks read back. |
start(cfg, session) | on the fresh VM | Stage the workspace: create dirs, write the input/ files and helper scripts. |
evaluate(cfg, session) | after the agent | Score the output against the reference; return list[float] (usually [score] in [0, 1]). |
All sandbox I/O goes through session (run a command, read/write
a file, screenshot, …). The full surface is
docs/SESSION_API.md.
reference/
during start(). The framework stages it only at eval time. End
start() with a "reference correctly hidden" check.
Never raise from evaluate() on missing output:
return [0.0]; raise only on infrastructure failures.
Other conventions: keep start() idempotent (wipe
output/ + reference/ first, in case of a reused
VM); put per-variant data in a VARIANTS list and read it from
cfg.metadata; use \r\n in expected text for Windows
tasks.
Shipping task data
| Pattern | For | How |
|---|---|---|
Inline in start() | prompts, small JSON, scripts (< a few hundred KB) | Just session.write_file(…), what demo/hello does, zero image work. |
| Baked into the image | datasets, checkpoints, large binaries | Place input/ + software/ + an encrypted reference.7z under <task_data_root>/<domain>/<task>/<variant>/; set task_data_source: baked_in_sandbox (or a gs:// bucket to stage per run). |
Discover & test
No registry. The loader picks up any tasks/<domain>/<task>/main.py
with the three hooks. Verify and smoke-test:
uv run python -m ale_run list | grep <domain>/<task>
uv run python -m ale_run run my_exp.yaml --task <domain>/<task> --dry-run
uv run python -m ale_run run my_exp.yaml --task <domain>/<task>When it's stable, add it to a curated list under selected_tasks/ and open a PR.