Configure an experiment

An experiment combines agent presets, one environment profile, and a task list. The experiment YAML holds run-level controls; provider and agent details stay in reusable config files.

Start from the example

cp example_exp.yaml my_experiment.yaml

name: my_experiment
secret_file: secret/.env

agents:
  - configs/agents/claude_code.yaml
environment: configs/environments/environment_gcloud.yaml
tasks: selected_tasks/hello_both.txt

output:
  root: .logs/ale
concurrency: 1
wall_time_s: 7200
cleanup_mode: delete

Relative paths resolve from the experiment file's directory. Keep the file in the repository root when using the shipped paths unchanged.

Experiment fields

Choose an environment profile

A task card requests a logical snapshot. Common values are:

Task cards normally supply a GCE machineType. The schema also accepts explicit vcpus and memory_gb values. Providers translate that shape to GCE, Docker limits, or QEMU guest resources. Environment-level overrides can cap local providers.

Choose a task list

Difficulty-specific lists live under selected_tasks/unlicensed/ and selected_tasks/full/. A text list contains one task path per line. A YAML list can select explicit variant indices.

Choose agent presets

Agent presets live under configs/agents/. Each file selects a harness, model, optional executor, and harness-specific configuration:

harness: claude_code
model: <model-id>
executor: sandbox
config:
  provider: openrouter
  max_turns: -1

The provider inside an agent's config is the LLM API provider. It is unrelated to the sandbox provider selected by the environment profile.

List multiple presets under agents: to run the same task list for each agent. Set an explicit agent id inside a preset when two configurations use the same harness and need separate result folders.

Configure credentials

secret/.env normally holds agent API keys and any variables referenced as ${env:NAME} by the selected environment:

OPENROUTER_API_KEY=...
ALE_REFERENCE_ARCHIVE_PASSWORD=rdi-ucberkeley-Gov8EV7wGHYAc7XQBzhd
GCP_PROJECT=...
GCP_SA_KEY=secret/gcp_key.json

ALE_REFERENCE_ARCHIVE_PASSWORD is required when the selected environment uses task_data_source: baked_in_sandbox. It is a public benchmark-wide archive key, not a private credential. ALE reads it on the host after agent execution and intentionally does not forward it into the agent sandbox environment or embed it in runtime source copied there.

Start by copying secret/.env.example to secret/.env, then fill in the API and provider credentials needed by your run. GCP variables are required only by the Google Cloud profile. Local profiles still need whichever API key their agent preset uses. Judge-based tasks can also require evaluator keys under secret/eval_time/.

Configure task data and output

These fields live in the environment profile because their implementation depends on the sandbox substrate:

task_data_source: baked_in_sandbox
output_path: local

# Required when the sandbox reads from or writes to GCS:
# gcs_sa_key: secret/gcp_key.json

• baked_in_sandbox uses encrypted references already stored in the image.
• gs://... stages data inside the sandbox with gsutil.
• local:... is currently implemented for the direct Docker provider.
• hf://... is accepted by configuration but task-data staging is not implemented yet.

Output behavior is covered in Run and collect results.