Terminal-Bench v1.0 & v2.0 Benchmark
- Terminal-Bench is a benchmark suite evaluating language agents on realistic terminal tasks using containerized environments and outcome-based tests.
- v1.0 featured manually curated tasks and binary verification scripts, while v2.0 introduced standardized containerization, unified harnesses, and rigorous audit protocols.
- The benchmark spans diverse workloads including system administration, scientific computing, and software engineering to assess agent performance in live shell environments.
Terminal-Bench is a benchmark family for evaluating language agents in real command-line environments, with an emphasis on long-horizon, economically valuable tasks that require execution, state tracking, tool use, and verifier-grounded completion rather than text-only answer production. In the literature, Terminal-Bench v1.0 appears as an earlier manually curated benchmark used as a primary target in early terminal-agent studies, while Terminal-Bench 2.0 is the formally published expansion that standardizes containerized tasks, human-written oracle solutions, outcome-based tests, Harbor as a unified harness, and Terminus 2 as a neutral baseline scaffold (Merrill et al., 17 Jan 2026, Zhu et al., 6 Feb 2026).
1. Purpose and research scope
Terminal-Bench was introduced to measure whether agents can perform the kind of work practitioners carry out in terminals: software engineering, scientific computing, cybersecurity, machine learning, system administration, and data wrangling. The benchmark centers on the terminal as “a ubiquitous, versatile, and powerful interface,” and it evaluates tasks that require chaining commands, manipulating files, configuring services, invoking compilers and interpreters, and producing verifiable changes in the environment rather than merely generating plausible text (Merrill et al., 17 Jan 2026).
This orientation distinguishes Terminal-Bench from benchmarks that focus on static code generation, browser interaction, or short-answer reasoning. The core object of evaluation is not an isolated command or a canned script, but an agent embedded in a live shell with side effects, partial observability, and long temporal dependencies. Later terminal-agent papers consistently use Terminal-Bench as the main external benchmark for synthetic-environment generation, supervised fine-tuning, reinforcement learning, harness optimization, and context-compression methods, which indicates that it has become the de facto shared evaluation substrate for this research area (Pi et al., 24 Feb 2026, Ren et al., 21 Apr 2026, Cheng et al., 20 May 2026, Wang et al., 15 Jul 2026).
2. Task model, environments, and verification
The benchmark’s task model is containerized and outcome-oriented. In Terminal-Bench 2.0, each task includes an instruction, a Docker image or Dockerfile-based environment, tests, an oracle solution, and a time limit; agents operate inside the same container that is later used for verification (Merrill et al., 17 Jan 2026). The official design explicitly separates execution from evaluation: tests verify properties of the final container state, not the agent’s command trace or console output. This makes the benchmark tolerant to multiple valid solution paths while remaining strict about correctness (Merrill et al., 17 Jan 2026).
A closely related formulation appears in v1.0 evaluations. In TermiGen’s description of TerminalBench 1.0, each task runs inside an isolated Docker container, verification is performed with “a dedicated verification script (test.sh),” and “an agent is considered successful only if it passes all test cases defined in the verification script” (Zhu et al., 6 Feb 2026). Although the later literature does not provide a full official v1.0 specification, these descriptions show continuity in the benchmark’s core principles: executable environments, per-task verification, and pass/fail scoring tied to actual terminal state.
This design addresses two persistent difficulties in terminal-agent evaluation: executability and verifiability. Executability matters because each task may require its own software stack, services, or data artifacts. Verifiability matters because correct outputs are heterogeneous: created files, repaired repositories, running services, numerical results, or configuration changes. Terminal-Bench’s reliance on executable tests over final state became a template for subsequent training-data pipelines such as TerminalTraj, TermiGen, LiteCoder-Terminal, Terminal-World, and SETA, all of which adopt Dockerized environments and verifier-grounded supervision in response to the benchmark’s requirements (Wu et al., 1 Feb 2026, Zhu et al., 6 Feb 2026, Peng et al., 28 May 2026, Cheng et al., 20 May 2026, Shen et al., 12 Jul 2026).
3. Terminal-Bench v1.0
In the available sources, Terminal-Bench v1.0 is primarily documented through evaluation papers rather than through a standalone official benchmark paper. TermiGen states that TerminalBench required “extensive manual curation to produce merely 200 tasks,” and uses TerminalBench 1.0 as its primary benchmark (Zhu et al., 6 Feb 2026). The same paper characterizes the evaluated tasks as realistic terminal workflows in isolated Docker containers, spanning build systems, system administration or DevOps, data processing and ETL, MLOps, algorithms, software debugging, scientific computing, interactive environments, distributed or parallel computing, and formal verification or graphics (Zhu et al., 6 Feb 2026).
The v1.0 evaluation protocol, as described there, reports “Average Pass@1 across three independent runs” with temperature 0.6 for fine-tuned and base models. Success is binary at the task level: the verifier script must pass all test cases (Zhu et al., 6 Feb 2026). This protocol made v1.0 a natural target for early work on terminal-agent training data. TermiGen-Qwen2.5-Coder-32B reaches “31.3 ± 1.8%” average pass rate on TerminalBench 1.0; TerminalTraj-32B reports 35.30% on TB 1.0; and LiteCoder-Terminal-32b-sft reports 29.06% pass@1 on Terminal Bench 1.0 (Zhu et al., 6 Feb 2026, Wu et al., 1 Feb 2026, Peng et al., 28 May 2026).
Several later papers continue to evaluate on v1.0, but they also state that they do not provide a formal side-by-side specification of v1.0 against v2.0. TerminalTraj treats both versions as hard, realistic terminal benchmarks and reports that v2.0 is stricter, while LiteCoder-Terminal repeatedly treats v2.0 as harder than v1.0 without enumerating a formal change log (Wu et al., 1 Feb 2026, Peng et al., 28 May 2026). This suggests that v1.0 should be understood as the benchmark’s earlier large manually curated release, but not as a version whose official design is fully reconstructed by the presently cited secondary sources.
4. Terminal-Bench 2.0
Terminal-Bench 2.0 is the benchmark’s formally published version and is described in detail in the benchmark paper. It is “a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows,” and each task has “a unique environment, human-written solution, and comprehensive tests for verification” (Merrill et al., 17 Jan 2026). The benchmark paper further states that 229 crowd-sourced tasks were contributed and 89 were selected after a multi-stage auditing pipeline (Merrill et al., 17 Jan 2026).
The 2.0 release expands rigor as much as scale. Harbor provides a unified task format and execution harness, and the release is runnable via harbor run -d [email protected] (Merrill et al., 17 Jan 2026). Terminus 2 serves as a minimal, neutral scaffold for headless terminal interaction through Bash, with support for interactive editors, key events such as Ctrl-C, and context summarization on long runs (Merrill et al., 17 Jan 2026). Verification is reinforced by automated solvability checks through oracle solutions, contributor checklists, deterministic CI checks, LLM-backed check tools, an adversarial exploit agent, and multi-auditor manual review (Merrill et al., 17 Jan 2026).
The benchmark paper also characterizes 2.0 as unsaturated for frontier systems at publication time: frontier models and agents score less than 65%. The top overall result in the benchmark paper is Codex CLI with GPT-5.2 at 62.9% ± 3.0%, while the best open-weight model there is Terminus 2 with Kimi K2 Thinking at 35.7% ± 2.8% (Merrill et al., 17 Jan 2026). Human time estimates underscore task difficulty: among experts, 48.6% of tasks were estimated to take under 1 hour and 47.3% between 1 and 24 hours; among juniors, 71.6% were estimated at 1–24 hours, 16.2% at 1–7 days, and 4.1% above 1 week (Merrill et al., 17 Jan 2026).
A concise comparison between the two versions, restricted to what the cited literature explicitly states, is as follows.
| Dimension | v1.0 | v2.0 |
|---|---|---|
| Size stated in cited sources | “200 tasks” via extensive manual curation (Zhu et al., 6 Feb 2026) | “89 tasks” selected from 229 crowd-sourced tasks (Merrill et al., 17 Jan 2026) |
| Execution environment | Isolated Docker container (Zhu et al., 6 Feb 2026) | Harbor task format with per-task containerized environments (Merrill et al., 17 Jan 2026) |
| Verification | Dedicated test.sh; success only if all test cases pass (Zhu et al., 6 Feb 2026) |
Outcome-based tests over final container state; commands and console output are not scored (Merrill et al., 17 Jan 2026) |
| Standard scaffold | Later papers use BashAgent or Terminus-2 (Zhu et al., 6 Feb 2026, Wu et al., 1 Feb 2026) | Harbor plus Terminus 2 neutral baseline (Merrill et al., 17 Jan 2026) |
The literature does not provide a complete official v1.0-to-v2.0 change log. Even so, the published 2.0 documentation clearly shows a benchmark that is more standardized, more heavily audited, and more explicit about harness comparability and verification discipline than the v1.0 descriptions available through later evaluation papers.
5. Metrics, harnesses, and reporting conventions
Terminal-Bench uses verifier-grounded success metrics, but the reporting convention varies across papers. In the benchmark paper, the central aggregate is resolution rate, reported with 95% confidence intervals (Merrill et al., 17 Jan 2026). In v1.0 evaluations such as TermiGen, the central metric is pass rate, defined operationally by full success on the verification script, and results are reported as “Average Pass@1 across three independent runs” (Zhu et al., 6 Feb 2026).
For Terminal-Bench 2.0, several later papers report pass@k. Terminal-World gives the explicit formula
with success judged by exact-match verification over the post-execution environment state (Cheng et al., 20 May 2026). TerminalTraj, by contrast, reports overall success as
and studies test-time scaling with pass@k as the probability that at least one of sampled trajectories solves the task (Wu et al., 1 Feb 2026).
The harness layer is likewise important. Official 2.0 uses Harbor and Terminus 2 (Merrill et al., 17 Jan 2026). Nemotron-Terminal describes a Terminus 2 response schema with JSON fields analysis, plan, commands, and optional task_complete, where commands are verbatim keystroke strings plus durations and special keys use tmux-style escapes (Pi et al., 24 Feb 2026). Terminal-World evaluates most models under a standardized Terminus2 controller with the same analysis–plan–commands structure, designed to reduce differences caused by bespoke controllers (Cheng et al., 20 May 2026). Continual-learning work on Terminal-Bench 2.0 uses Harbor v0.16.1 and defines a repeated-trial pass-rate metric
$\text{PassRate}(A, \mathcal{T}) = \frac{1}{R\,|\mathcal{T}|}\sum_{t \in \mathcal{T}} \sum_{r=1}^{R} \mathbbm{1}[A \text{ solves } t \text{ on trial } r],$
with derived Phase-1, Transfer, Final, and LifelongAvg scores on a hard-task split (Wang et al., 15 Jul 2026).
These variants do not alter the benchmark’s central invariant: correctness is judged by executable verification over environment state. What changes across papers is whether the reported number is single-try success, pass@k, repeated-trial average, or a derived continual-learning aggregate. For Terminal-Bench, metric interpretation is therefore inseparable from harness and protocol specification.
6. Research role, empirical landscape, and open issues
Terminal-Bench rapidly became the organizing benchmark for terminal-agent research. Synthetic-data systems use it as the principal downstream target: TerminalTraj builds 32,325 Docker images and 50,733 verified trajectories, reaching 35.30% on TB 1.0 and 22.00% on TB 2.0 with TerminalTraj-32B (Wu et al., 1 Feb 2026). Nemotron-Terminal studies data engineering for TB2.0 and reports 13.0 ± 2.2 for 8B, 20.2 ± 2.7 for 14B, and 27.4 ± 2.4 for 32B under Terminus 2 (Pi et al., 24 Feb 2026). Terminal-World constructs 5,723 training environments and reports 31.5 Pass@1 and 43.8 Pass@3 on Terminal-Bench 2.0 for Terminal-World-32B, surpassing Nemotron-Terminal-32B by +4.5 Pass@1 under its reported setup (Cheng et al., 20 May 2026). SETA, a reinforcement-learning environment-generation framework, trains Qwen3-8B with GRPO on SETA-Env and achieves 12% pass rate on Terminal-Bench 2.0, while also improving DeepSeek-V4-Flash from 40% to 43% pass@1 and from 54% to 58% pass@5 under the same terminal-agent harness (Shen et al., 12 Jul 2026).
The benchmark also supports research beyond data generation. TACO treats Terminal-Bench as the canonical testbed for terminal-context compression and reports consistent gains on both TB 1.0 and TB 2.0, including +2.88 points for Qwen3-32B on TB 1.0 and +3.56 points on TB 2.0, while reducing per-step token overhead for large models by around 10% (Ren et al., 21 Apr 2026). A continual-learning study constructs a two-phase protocol from the hard subset of Terminal-Bench 2.0 and finds that only RELAI-VCL compounds optimization gains across phases, achieving a LifelongAvg of 76.4% versus 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline on its 22-task hard split (Wang et al., 15 Jul 2026).
Several recurring misconceptions are corrected by the benchmark literature. First, Terminal-Bench is not merely a coding benchmark: it spans software engineering, scientific computing, security, ML engineering, system administration, and related workflows (Merrill et al., 17 Jan 2026). Second, it does not reward a preferred command sequence; it rewards an acceptable final state (Merrill et al., 17 Jan 2026). Third, cross-version comparisons are possible only with caution. Multiple papers evaluate both v1.0 and v2.0 and report materially lower scores on v2.0 for comparable backbones, and some explicitly describe v2.0 as harder; this suggests increased difficulty and audit rigor, but the exact official v1.0-to-v2.0 delta is not fully documented in the presently cited literature (Wu et al., 1 Feb 2026, Peng et al., 28 May 2026).
The benchmark’s limitations are likewise explicit. Terminal-Bench 2.0 allows internet access for realism, which can introduce external variability; no private held-out set is used, though canary strings are included to discourage contamination; and some residual non-determinism remains because hardware resources, network stability, and long-running tasks can vary across executions (Merrill et al., 17 Jan 2026). The benchmark paper also notes a saturation risk for 2.0 and states an intention to release additional challenging task sets (Merrill et al., 17 Jan 2026). In that sense, Terminal-Bench v1.0 and v2.0 should be understood not simply as fixed scoreboards, but as successive attempts to define a robust evaluation regime for agents acting in real command-line environments.