Run an experiment

Choose where task sandboxes run, connect an agent and task list, validate the matrix, then launch. The provider determines operating-system support, isolation, setup cost, and practical concurrency.

Install ALE

git clone git@github.com:rdi-berkeley/agents-last-exam.git
cd agents-last-exam
uv sync --all-packages

Choose a sandbox provider

ALE exposes four provider implementations. The names below describe the sandbox the task receives. The code identifier used in environment YAML is shown in parentheses.

Provider	Sandbox	Current coverage	Best use
Google Cloud VMs `gcloud`	A fresh GCE VM for every run unit	145 tasks in `unlicensed.txt`. The seven additional tasks in `full.txt` require your own licensed-software activation.	Production benchmark runs and elastic concurrency
AWS EC2 `aws`	A fresh EC2 instance for every run unit	Linux + Windows from public AMIs. GPU tasks not supported.	Production runs on AWS instead of/alongside GCP
Alibaba Cloud ECS `aliyun`	A fresh ECS instance for every run unit	Linux + Windows from the same image families. GPU tasks not supported.	Production runs on Alibaba Cloud instead of/alongside GCP
Local containers `docker`	A fresh Linux container for every run unit	99 tasks in `docker_support.txt`	The lightest local path for the supported Linux subset
QEMU/KVM VMs `qemu`	A fresh QEMU/KVM guest VM for every run unit	140 CPU-compatible Ubuntu and Windows tasks in `cpu_unlicensed.txt`	Full-OS local testing, including Windows, without cloud VMs
Existing sandbox `static`	One CUA-enabled machine you start and manage	Whatever that machine's OS, software, and data support	Image development, debugging, and reproducing one task

QEMU/KVM is one local-VM implementation

Each task runs inside a complete QEMU/KVM guest. Docker packages and supervises the VM runner, but it is not the task sandbox. The runner inherits its QEMU, networking, and noVNC stack from CUA and Dockur. The provider-specific name leaves room for future local VM backends, such as VMware, as sibling providers. The stable ALE interface remains provider: qemu.

Match the task list to the provider

A task requests a logical snapshot such as cpu-free-ubuntu, cpu-free, or gpu-free. The selected environment must map every requested snapshot to a provider. Use the provider-specific allowlists for local runs:

Environment	Recommended task list
`configs/environments/environment_gcloud.yaml`	`selected_tasks/unlicensed.txt`, or `full.txt` after licensed software is activated
`configs/environments/environment_aws.yaml`	`selected_tasks/unlicensed.txt` (same image families on AWS EC2)
`configs/environments/environment_aliyun.yaml`	`selected_tasks/unlicensed.txt` (same image families on Alibaba Cloud ECS)
`configs/environments/docker.yaml`	`selected_tasks/docker_support.txt`
`configs/environments/qemu.yaml`	`selected_tasks/cpu_unlicensed.txt`
A static environment	A small list known to match the attached machine

The run workflow

Follow the provider guide and verify its prerequisites.
Configure an experiment with one environment, one or more agents, and a compatible task list.
Run --dry-run to inspect the exact agent × task matrix.
Launch, resume, and collect results.

Supported

Google Cloud VMs

The broadest coverage and the production path for batch runs.

Supported

Alibaba Cloud

Ephemeral ECS instances + OSS. Linux + Windows; GPU tasks not supported.

Supported

Local containers

A lighter local Linux sandbox with host-staged task data.

Supported

AWS

Ephemeral EC2 instances + S3. Linux ready; Windows is BYOL + dedicated.

Supported

QEMU/KVM VMs

CPU-only Ubuntu and Windows guests through QEMU/KVM.

Advanced

Existing sandbox

Attach ALE to a CUA endpoint you already operate.