Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

The problem

AI agents are increasingly given a terminal and asked to get work done inside it: installing software, moving files around, fixing a broken configuration, running a build until it passes. Judging progress on such work is awkward. A chat-style question has an answer that can be compared with a reference answer; a terminal session has a starting state, a mess of intermediate commands, and an end state that may or may not be correct. Terminal-Bench was built to supply that missing measurement.

What the benchmark contains

The paper presents Terminal-Bench 2.0, a carefully curated hard benchmark made up of 89 tasks set in computer terminal environments, inspired by problems that come up in real workflows^[1]. Curation matters here: the tasks were selected rather than scraped, so each one is meant to represent something a person might actually have to do at a command line.

Every task comes with its own unique environment, a human-written solution, and comprehensive tests used to verify the result^[1]. In structural terms, a task consists of an instruction, a Docker image, a set of tests, an example solution and a time limit^[2]. The Docker image fixes the starting state so that the task behaves the same way on every machine; the instruction is what the agent sees; the tests decide whether the job was done; the example solution shows that a solution exists and gives a reference point; and the time limit keeps the evaluation bounded.

How a task is scored

Because the environment is a container, evaluation can be automatic. The agent is placed in the image, reads the instruction, and works through the terminal until it believes the task is finished. The tests are then run against the resulting state of the container. A task counts as solved when those tests pass; the example solution indicates that this is achievable within the limits given. This design keeps grading objective and reproducible, and avoids relying on a human to read a transcript and judge whether the work looks right.

Results

The headline result is a large gap between current systems and full success. The authors show that frontier models and agents score less than 65% on the benchmark^[1]. Smaller models fare much worse, scoring around 15%^[2]. The benchmark therefore separates systems that a simpler test might treat as equivalent: even the strongest models and agent scaffolds leave a substantial share of tasks unsolved.

Beyond the raw numbers, the authors carry out an error analysis to identify areas where models and agents could be improved^[1]. This is the more useful half of a benchmark paper. A single accuracy figure says how much is missing; an analysis of the failures says what kind of capability is missing, which is what a developer needs in order to decide what to change.

Why it matters

The dataset and the evaluation harness are published openly, so that developers and researchers can use them in future work^[1]. Openness matters for a benchmark of this kind on two counts. It lets groups compare their systems on the same tasks instead of on private test sets, and it lets them inspect the tasks and the grading, which is the only way to trust a number.

Terminal work is also a demanding setting in its own right. It requires reading unfamiliar systems, acting in sequences where an early mistake propagates, and knowing when a result is actually correct rather than merely plausible. A benchmark built from curated, realistic tasks with executable verification gives that kind of capability something to be measured against, and gives the field a shared target to aim at.

Provenance

Mike A. Merrill, Alexander G. Shaw and Nicholas Carlini led a large author list that also includes Ludwig Schmidt, Andy Konwinski and Alex Dimakis. The paper was submitted to arXiv on January 17, 2026^[1].