Long-Horizon Terminal Bench

• Long-Horizon Terminal Bench is a benchmark designed to assess AI agents on multi-stage, long-horizon workflows using dense, subtask-based grading.
• It replaces binary end-state evaluation with a deterministic scoring system that rewards partial progress across complex, containerized terminal tasks.
• The benchmark is implemented in Harbor format and highlights practical challenges in planning, debugging, and sustained task execution over prolonged periods.

Searching arXiv for the benchmark and closely related terminal-agent benchmark papers to ground the article in current literature. arXiv_search(query="Long-Horizon-Terminal-Bench", max_results=5) Long-Horizon-Terminal-Bench, often abbreviated LHTB, is a terminal benchmark for evaluating AI agents on genuinely long-horizon, multi-stage workflows rather than short command-line tasks with binary end-state grading. It was introduced to address a gap in prior terminal-agent evaluation: existing benchmarks often emphasize tasks that finish in a few minutes, require only tens of steps, and are judged almost entirely by whether the final output passes a test suite. LHTB instead combines much harder tasks with deterministic dense reward-based grading, so that evaluation captures not only whether an agent reaches the final goal, but also how far it progresses through an open-ended workflow (Li et al., 9 Jul 2026).

1. Origin and problem framing

LHTB is positioned as a direct response to limitations in prior terminal-agent benchmarks such as Terminal-Bench 2.0 and related software-engineering evaluations. In those earlier settings, an agent that completes most of a complex pipeline but fails on a final hidden check is treated the same as one that failed immediately. The benchmark’s central intervention is therefore not only to make tasks harder, but to change the scoring philosophy from sparse outcome-only grading to dense, deterministic partial credit (Li et al., 9 Jul 2026).

Conceptually, the benchmark is designed around long-horizon behaviors that current agents often fail to sustain: planning over hundreds of steps, long-context management, iterative debugging, repeated repair of broken pipelines, recovery from mistakes, verification before termination, and effective time allocation before timeout. The paper contrasts this with short-horizon benchmarks, which mostly test whether an agent can act correctly locally. LHTB is intended to measure whether locally sensible actions accumulate into completed long-running workflows.

A common misconception is that LHTB is merely a larger Terminal-Bench-style task set. The benchmark keeps the Terminal-Bench-style environment model, but changes both the task scale and the evaluation target. This means the benchmark is about long-horizon completion under dense grading, not simply more difficult binary pass/fail terminal tasks.

2. Task structure, environment model, and benchmark scope

Every LHTB task is a containerized terminal environment in Harbor format with four core components: a natural-language instruction, a Docker image defining the environment, a task configuration file, and an oracle implementation or simulator used for grading (Li et al., 9 Jul 2026). The agent sees only the top-level instruction and must act entirely through the terminal by issuing shell commands, inspecting files, editing code, running scripts, interpreting outputs, and iteratively debugging.

The Docker image contains all code, data, tools, helper scripts, and assets needed to solve the task. In addition, each task is built as a complete but deliberately broken terminal-only project and includes public asset-generation scripts, a weak baseline implementation, an official gold solution, a multi-step solve.sh, and a hidden verifier. The official gold solution must achieve verifier score $1.0$ on the hidden suite.

The benchmark contains 46 tasks. The main text states that they span nine categories, while the appendix provides a finer-grained taxonomy and notes that the tasks cover 21 high-level domains. The nine-category taxonomy used throughout the paper includes: Interactive games; Research reproduction / ML; APEX professional workflows; Software / reverse engineering; Multimodal / imaging analysis; Systems / performance / security; Earth, climate / energy; Scientific computing / simulation; and Logic / constraint puzzles.

Representative tasks include 2048, chess-mate, super-mario, snake_maze_campaign, sokoban, rush_hour_campaign, duckdb-optimizer-closure, grammar-fuzz-coverage-hunt, spot-scheduler-traces, vector-db-iterative-build, robotics-slam-benchmark-repair, climate-netcdf-extreme-event-audit, dicom-radiology-audit, document-table-layout-reconstruction, scientific-figure-data-reconstruction, riscv-core-debug, foldseek-paper-reproduction, and unison-paper-reproduction. This breadth is meant to stress open-ended workflows rather than one-shot command generation.

3. Dense reward-based grading

The benchmark’s central methodological contribution is its subtask-based dense grading. Each task is decomposed into a small set of semantically meaningful subtasks,

$\{s_1,\dots,s_K\},$

intended to reflect intermediate goals in the intended workflow. For each subtask $s_k$, the deterministic grader returns a normalized subscore

$r_k \in [0,1].$

The overall task reward is the weighted average

$$R \;=\; \frac{\sum_{k=1}^K w_k\, r_k}{\sum_{k=1}^K w_k},$$

where $w_k$ are non-negative weights. By default, weights are equal, though the final goal can be given a higher weight when appropriate (Li et al., 9 Jul 2026).

The paper distinguishes three kinds of subtasks. Binary subtasks return $r_k \in \{0,1\}$ from strict programmatic checks such as whether unit tests pass or a service responds on the expected port. Continuous or thresholded subtasks assign partial credit for quantitative targets, including metric reproduction error or fraction of held-out examples matching oracle predictions. Episode-aggregating subtasks are used for campaign or repeated-trial settings such as games and audits, where the score may be the fraction of episodes or levels solved, or the mean normalized simulator reward across runs.

This reward is computed deterministically from the final container state after a rollout ends. The grader uses objective evidence such as files, generated outputs, test results, or simulator state. A task is counted as solved when

$R \ge \tau,$

for a success threshold $\tau$. The benchmark reports pass@1 at reward thresholds $0.9$, $\{s_1,\dots,s_K\},$0, and $\{s_1,\dots,s_K\},$1, with $\{s_1,\dots,s_K\},$2 representing full completion. Mean normalized reward across tasks is also reported to capture progress on unsolved tasks.

A second misconception is that dense grading implies judge-model subjectivity. In LHTB, grading is deterministic and environment-grounded rather than based on model judgment. This makes near-misses, partial progress, and verification failures visible without abandoning programmatic evaluation.

4. Benchmark construction and verification regime

The benchmark construction process begins from realistic professional or scientific workflows, including materials phase-diagram auditing, robotics SLAM repair, climate NetCDF extreme-event detection, audio-visual event alignment, scientific figure data reconstruction, satellite flood change detection, microscopy cell-count quality control, and scanned-document table reconstruction (Li et al., 9 Jul 2026). Tasks are then converted into deliberately broken terminal-only projects.

A central design feature is the hidden verifier. Public checks are intentionally low-weight and mostly validate command-line behavior, file formats, and a few simple examples, while most reward comes from hidden stress cases that vary schema or inputs. The paper lists examples such as nested manifests, gzip+base64 wrappers, missing values, renamed fields, injected noise, rotated/cropped images, anomalous frames, and alternate coordinate or time conventions. This is meant to prevent overfitting to visible tests and to force robust implementations.

For difficulty calibration, the authors generated 120 candidate tasks and filtered them down to the final 46. They repeatedly ran DeepSeek V4 Pro under 1.5-hour budgets and adjusted task design until tasks were “challenging but still solvable in principle.” All final tasks are implemented in Harbor format with containerized environments and a shared evaluation harness. The benchmark and evaluation harness are released, with a project page at https://zli12321.github.io/LHTB/.

The appendix also labels task difficulty from average task reward: tasks with mean reward at least $\{s_1,\dots,s_K\},$3 are labeled Easy, otherwise Hard. Most tasks are Hard. Easy-labeled examples include apex-ib244-matter, apex-law433-matter, generals-bot-arena, grammar-fuzz-coverage-hunt, nbody-accel-iterative, poc-exploit-craft, riscv-core-debug, and spot-scheduler-traces. Hard examples include most multimodal audits, scientific regressions, puzzle campaigns, experiment reproductions, and game-playing tasks.

5. Experimental protocol and headline results

LHTB is evaluated using the Harbor framework with the Terminus-2 agent harness, except GPT-5.3, which was evaluated using Codex as the harness. The model set contains 15 frontier models: GPT-5.5, GPT-5.4, GPT-5.3 Codex, DeepSeek V4 Pro, Gemini 3.1 Pro, GLM 5.1, GLM 5.2, Kimi K2.6, Kimi K2.7 Code, MiniMax M3, Qwen3.7 Max, Qwen3.6 Plus, Doubao Seed 2.1 Pro, Hy3, and Grok 4.20 (Li et al., 9 Jul 2026).

The reported metrics are pass@1, mean normalized reward, episodes per task, time per task, token usage per task, and estimated dollar cost. In this paper, pass@1 is the fraction of tasks solved by a single rollout per task, where solved means reward above a threshold such as $\{s_1,\dots,s_K\},$4 or $\{s_1,\dots,s_K\},$5.

The resource profile is one of the benchmark’s defining properties. Across 15 models, rollouts average about 228–231 episodes per task, 9.66M–9.9M tokens per task, and 85.1–85.3 minutes wall-clock time, under a 90-minute timeout. This is contrasted explicitly with Terminal-Bench 2.0, where tasks average roughly 20–30 minutes and 20–30 episodes per task. The paper summarizes LHTB as “an order of magnitude more demanding than prior terminal-based benchmarks.”

The quantitative results are low across the board. GPT-5.5 is the strongest model, but reaches only 15.2% pass@1 at $\{s_1,\dots,s_K\},$6, corresponding to 7/46 tasks, and 10.9% at $\{s_1,\dots,s_K\},$7. The mean pass rate across all 15 models is 4.3% at $\{s_1,\dots,s_K\},$8 and 1.7% at $\{s_1,\dots,s_K\},$9. MiniMax M3, Kimi K2.7 Code, and DeepSeek V4 Pro follow at 6.5%. Kimi K2.6 and Grok 4.20 solve zero tasks at $s_k$0, and Grok 4.20 has the lowest mean reward, $s_k$1.

A notable result is that at $s_k$2, 10 of the 15 models solve zero tasks. The paper therefore emphasizes mean reward as a secondary signal, and reports only moderate rank correlation between pass rate and mean reward, with Spearman

$s_k$3

This suggests that full completion and sustained partial progress capture different aspects of long-horizon competence.

6. Failure modes, reward distributions, and interpretive claims

The reward-distribution analysis motivates the dense-grading design. Across all $s_k$4 model-task runs, only 30 runs (4.3%) pass the $s_k$5 threshold, while 227 runs (32.9%) achieve almost no progress with $s_k$6. The remaining 433 runs (62.8%) achieve nontrivial partial reward but would be treated as failures under binary grading. Moreover, 180 runs (26.1%) reach $s_k$7, and near-miss runs with

$s_k$8

occur 73 times, more than twice the number of passes (Li et al., 9 Jul 2026).

The paper highlights Kimi K2.6 as an illustrative case: it solves zero tasks at $s_k$9, yet has five near-misses and mean reward $r_k \in [0,1].$0, including a best run at $r_k \in [0,1].$1 on grammar-fuzz-coverage-hunt. This is presented as evidence that binary grading collapses meaningful differences in long-horizon progress.

The dominant failure mode is timeout. Among unresolved runs with $r_k \in [0,1].$2, 79% (518/660) end because the 90-minute budget expires while the agent is still actively working. Only 19% are early exits, and 3% are harness errors. Timed-out runs are usually not almost complete: their mean reward ranges only from 0.10 to 0.35 across models. The paper interprets this to mean that the main bottleneck is not a final small fix, but failure to sustain coherent progress quickly enough.

A second failure mode is false finish, caused by weak self-verification. Among 124 runs where the agent voluntarily exited before timeout, the authors identify 14 false finishes with

$r_k \in [0,1].$3

Examples include Kimi K2.7 Code stopping on duckdb-optimizer-closure at $r_k \in [0,1].$4, GLM 5.2 stopping on apex-ib244-matter at $r_k \in [0,1].$5, and seven models stopping on apex-law433-matter between $r_k \in [0,1].$6 and $r_k \in [0,1].$7, often with about 20 minutes remaining. The benchmark uses dense rewards to expose these as verification failures rather than undifferentiated non-solutions.

The paper also notes that stronger performance is not simply a function of spending more tokens or money. GPT-5.4, for example, is the most expensive model in the cost table yet performs worse than GPT-5.5, partly because it uses many more episodes.

7. Position within terminal-agent research

LHTB is best understood as an extension of the Terminal-Bench line of evaluation. Terminal-Bench 2.0 defines hard, realistic command-line tasks in Harbor-format containerized environments and scores them by final environment-state correctness under binary outcome-based tests (Merrill et al., 17 Jan 2026). LHTB preserves the terminal-only environment model but shifts to much longer tasks and dense intermediate grading, thereby turning partial progress into a measurable quantity rather than a discarded failure state.

Several adjacent works target bottlenecks that LHTB makes especially visible. “Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents” reports pass@1 gains of +8.3 pp on Terminal-Bench 2.0 for a Sonnet 4.5 action agent by adding a selective memory sidecar, framing the core failure as “behavioral state decay” (Wu et al., 9 Jul 2026). “A Self-Evolving Framework for Efficient Terminal Agents via Observational Context Compression” proposes TACO, which improves TerminalBench 1.0 and 2.0 scores while reducing token overhead by around 10% on many settings (Ren et al., 21 Apr 2026). “CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents” trains compacting agent policies and reaches 24.5% Pass@1 on Terminal-Bench 2.0 with GLM-4.5-Air (Li et al., 6 Jul 2026). “LiteCoder-Terminal” and “KLong” address long-horizon training data and curriculum design for terminal-style or coding-heavy agents, with transfer gains on Terminal-Bench 2.0 and Terminal-Bench Hard (Peng et al., 28 May 2026, Liu et al., 19 Feb 2026).

A plausible implication is that LHTB functions as a stress test for precisely these research directions: long-context management, structured memory, iterative debugging, calibrated stopping, and environment-grounded self-verification. Its main contribution is therefore not only a harder task set, but a measurement regime that exposes how and where current terminal agents fail over extended workflows.