Run and collect results

Validate the matrix before provisioning anything, launch the experiment, resume interrupted sweeps, and choose how produced files leave the sandbox.

Validate before launch

uv run python -m ale_run run my_experiment.yaml --dry-run

The dry run resolves the experiment, agent presets, environment routing, and task list. It prints every agent × task × variant unit without creating a sandbox.

Launch and filter

uv run python -m ale_run run my_experiment.yaml

# Run only selected entries from the matrix:
uv run python -m ale_run run my_experiment.yaml \
  --agent claude_code \
  --task demo/hello

concurrency limits units in flight. Each active unit owns one sandbox while ALE provisions it, stages data, runs the agent, collects artifacts, and evaluates the result. Provider quotas, host memory, disk latency, and LLM rate limits can all become the practical bound.

Resume and retry failed units

auto_resume: true
max_attempts: 3

Auto-resume is enabled by default. It scans prior run.json files under the same experiment name and output root, skips units with status completed or timeout, and immediately requeues newly failed units. max_attempts accepts 1 through 3 and counts the initial run within that invocation. Every attempt keeps its own run directory.

Disable resume for one invocation

uv run python -m ale_run run my_experiment.yaml --disable-resume

This command-line override runs every selected unit exactly once. It does not scan prior results or retry failures. Set auto_resume: false in the experiment YAML to make that behavior persistent.

Choose where task output goes

output.root in the experiment controls the host directory for run records. output_path in the environment controls what happens to files produced inside the task's sandbox:

Value	Behavior	Use
`null`	Do not copy the task output before cleanup.	Score-only runs where produced files are unnecessary.
`local`	Copy into the run's `output/` directory.	Local development and outputs you want beside the run record.
`gs://<bucket>`	Upload from the sandbox to `<bucket>/<run_id>/output/`.	Large batch artifacts and centralized storage.

Local collection uses the fastest available provider transport: Docker uses docker cp, the QEMU provider uses a per-run guest share, and other providers use the CUA file API. GCS output uses sandbox-side gsutil. Managed providers use gcs_sa_key to inject a service-account key; a static sandbox must already have working Cloud Storage credentials.

Output collection is best-effort

An output transfer failure is recorded in events.jsonl but does not prevent evaluation while the sandbox is still alive. Check output_gather_done or output_gather_failed when artifacts are required.

Choose cleanup behavior

Mode	Managed providers
`delete`	Remove the GCE VM, Docker container, or QEMU container and disposable overlay.
`stop`	Stop the sandbox but preserve its provider-managed state for inspection.
`keep`	Leave it running.

The static provider never owns the attached machine. Its optional cleanup script is the only release action it performs.

Read the result

Each unit writes a timestamped run directory containing events.jsonl, run.json, eval_result.json, the normalized trajectory, native agent logs, screenshots, and any locally collected output. See Trajectories & artifacts for the complete layout.