Measure agents on real work, in real sandboxes.
Agents' Last Exam runs AI agents on long-horizon, economically valuable
tasks inside reproducible desktop sandboxes, then grades what they
produce against hidden references. This site explains how the
ale_run framework works and how to run, configure, and
extend it.
What ALE is
ALE evaluates AI agents on long-horizon, economically valuable, real-world work with verifiable outcomes: the professional tasks that move real industries, not abstract puzzles. Built with 250+ industry experts and organized by the O*NET / SOC 2018 occupational taxonomy, it spans 55 subdomains across 13 industry clusters and 1,000+ tasks, with around 150 in the public release. It is a living benchmark, and far from solved: across mainstream agents, the average pass rate on the hardest tier is just 2.6%. ALE is built less as one more leaderboard than as an instrument for closing the gap between benchmark scores and real economic impact.
What keeps it real
Most benchmarks compress the world into a tidy API. ALE does the opposite: the tasks are real production workflows, so the evaluation has to be real too.
How a run works
Rather than puppeteer an agent step by step, ALE drops the whole system into a sandbox with the task, lets it work to completion, and grades what it leaves behind. Every run pairs three swappable pieces:
The orchestrator then takes that pairing through a fixed lifecycle. Each stage below is a concept this site covers in depth, so the run itself doubles as a map of what follows:
-
1 · Provision the sandboxA provider creates or attaches a sandbox and waits until its control server answers.
-
2 · Stage the taskThe task's input data is injected onto the sandbox and the machine is set into its starting state, ready for the agent.
-
3 · Run the agentThe agent does the work, running to completion and driving the machine through both the terminal and the GUI, the way a person would. Any harness plugs in the same way.
-
4 · Grade the outputOnly now is the hidden reference staged. The task's grader scores the agent's output against it, by exact comparison or an LLM/VLM judge.
-
5 · Record everythingEach agent's native logs are normalized into one uniform trajectory, written next to the score, the event log, and the raw artifacts.
-
6 · Tear downThe provider deletes, stops, keeps, or detaches the sandbox according to the experiment's cleanup mode.
From here
Those six stages are the four concept pages above. When you want to actually run it, or build on it: