Task spec & data staging

Curated real-world inputs, a hidden reference (the golden artifacts an output is graded against), and a grader that can't be gamed. The input and reference data are what make an ALE task more than a prompt, and keeping the reference out of the agent's reach is also important to keep the benchmark unhackable.

A task is a cua-bench main.py

ALE adopts cua-bench's task spec directly. A task is a single main.py with three functions, each driving the sandbox through the cua-bench DesktopSession, a session the provider opens onto the running sandbox by talking to a control server inside it:

• load() declares the task and its variants.
• start(cfg, session) sets the sandbox up before the agent runs: stage the inputs, open the apps, put the machine in its starting state.
• evaluate(cfg, session) runs after the agent finishes: read its output and the reference, and return a score.

One main.py is one workflow plus one general grading logic. Variants reuse that logic over different concrete inputs; load() returns one entry per variant, and the public release ships only the base variant of each task.

Alongside that code, a task ships the two kinds of data those functions move, collected and validated by domain experts so that the inputs plus the prompt determine a gradeable result:

Staging the data, hiding the reference

Before the agent can touch anything, the framework injects the task's input onto the sandbox, writing its files to a known place on disk. The reference is injected the same way, but only after the agent has finished, in a separate phase, so while the agent works the answer is never on the machine to find.

Where those files come from is set by the environment's task_data_source, which has three implemented modes:

• Runtime pull (gs://ale-data-public) — the general case: each task's data is fetched onto the sandbox when its slot runs. A sandbox you build yourself uses this, and from a Google Cloud VM the pull is fast.
• Baked — ALE's published Google Cloud images and QEMU/KVM qcow2 disks arrive with every task's data already on the disk, so a run starts with nothing to fetch. The reference then sits on the machine all run, so it is baked as an encrypted archive (reference.7z) unpacked only at grading.
• Local mount (local:<dir>) — for the local-container provider. The host holds the data (fetched once from Hugging Face); the framework docker cps each task's input into the container before the agent and the reference in after. No encryption is needed: the reference is simply never inside the container while the agent runs — the same timing that hides it above.

The configuration schema also reserves hf://... for direct Hugging Face staging, but that task-data backend is not implemented yet. The local-container fetch script downloads from Hugging Face on the host, then uses the implemented local: path.

Grading

Once the reference is available, the task's evaluate hook reads the agent's output alongside it and returns a list of scores. Two styles of grader are used:

• Deterministic: exact or structural comparison (files, values, parsed content). Most tasks.
• Judge-based: an LLM or VLM scores outputs that need semantic or visual judgment (e.g. an OpenAI vision judge, a Gemini video judge).

The grader runs as part of the task slot, host-side, after the agent is gone, so even judge-based scoring never touches the agent's run.