Long-Horizon Terminal Bench (LHTB)

• LHTB is a benchmark that evaluates AI agents on long-horizon terminal tasks requiring sustained planning, debugging, and iterative refinements.
• The benchmark decomposes each task into fine-grained subtasks with dense rewards, moving beyond binary final outcomes for detailed analysis.
• It spans 46 tasks across nine domains, highlighting challenges in maintaining performance over extended workflows compared to short-term benchmarks.

Long-Horizon-Terminal-Bench (LHTB) is a benchmark for evaluating AI agents on long-horizon terminal tasks: tasks that require many sequential actions, sustained planning, debugging, verification, and recovery from mistakes over tens of minutes to hours, rather than the short, mostly one-shot terminal tasks used in earlier benchmarks. It follows a Terminal-Bench-style setup, but departs from final-outcome-only grading by decomposing each task into fine-grained graded subtasks that yield dense intermediate rewards and partial credit. The benchmark contains 46 tasks spanning nine categories, and its central claim is that long-horizon execution itself remains a bottleneck for frontier agents even when they perform strongly on shorter terminal workloads (Li et al., 9 Jul 2026).

1. Motivation and benchmark rationale

LHTB was introduced in response to two limitations of existing terminal benchmarks. First, prior benchmarks largely focus on simple problems that finish within minutes, which underestimates the difficulty of realistic workflows that require hundreds of steps, repeated inspection, long-context tracking, iterative refinement, and careful stopping decisions. Second, prior benchmarks are often graded only by the final outcome, so an agent that nearly completes a task receives the same score as one that makes no meaningful progress. LHTB addresses both issues by targeting long-running terminal workflows and by assigning dense intermediate rewards rather than only binary solved/unsolved labels (Li et al., 9 Jul 2026).

The abilities that LHTB is designed to stress are explicitly long-horizon: long-horizon planning and replanning, maintaining state over many steps, iterative debugging and repair, progress verification and stopping judgment, robustness to hidden test cases and changing inputs, and work over multimodal, scientific, engineering, and software artifacts. This design positions LHTB as a benchmark for open-ended terminal execution rather than short command-generation or localized code-editing tasks.

A notable implication is that LHTB measures not only whether an agent reaches a final goal, but also how far it progresses on a difficult workflow before stalling, timing out, or terminating incorrectly. This makes it diagnostically different from benchmarks whose signal is sparse and exclusively terminal.

2. Task composition and environment structure

LHTB contains 46 tasks spanning nine categories of long-horizon work. The paper also mentions a broader 21-category taxonomy in the task descriptions, but describes the benchmark itself at the paper level as nine domains. These domains include interactive games, experiment reproduction, software engineering / reverse engineering, multimodal analysis, scientific computing / simulation, Earth, climate, and energy workflows, systems and performance, logic / constraint puzzles, and security / exploit crafting (Li et al., 9 Jul 2026).

The task inventory is deliberately heterogeneous. Examples include 2048, chess, Snake, Super Mario, and Sokoban; ALP paper reproduction, Foldseek paper reproduction, UNISON paper reproduction, tabular covariate shift, and robotics SLAM benchmark repair; DuckDB optimizer closure, vector-db iterative build, RISC-V core debug, LangChain version migration, and unknown-config-semantics; scientific-figure data reconstruction, document table layout reconstruction, satellite flood change detection, microscopy cell-count QC, and audio-visual event alignment; and scientific or climate workflows such as materials phase diagram audit, MODFLOW, SPICE, SU2, MATPOWER, OPF regression, and climate NetCDF extreme-event audit.

Each task uses a Terminal-Bench-style formulation consisting of a natural-language instruction, a Docker container defining the environment, a task configuration file, and an oracle implementation or simulator used for grading. The benchmark additionally includes a weak baseline implementation, an official gold solution, a multi-step solve.sh, and a hidden verifier. To reduce overfitting to visible tests, the benchmark uses light public checks and heavy hidden checks. Hidden stress cases include nested manifests, gzip + base64 wrappers, renamed fields, missing values, injected noise, rotated/cropped images, anomalous frames, and alternate coordinate or time conventions. The authors report that they generated about 120 candidate tasks and filtered them to the final 46 (Li et al., 9 Jul 2026).

This construction makes LHTB distinctively terminal-centric. Agents interact only through the terminal by issuing shell commands, editing files, running scripts, inspecting outputs, and iterating until success or timeout. The tasks are intentionally structured so that a single task usually requires many steps and many intermediate decisions, not merely one or two edits or command invocations.

3. Dense reward formulation and grading methodology

The benchmark’s methodological novelty is its dense reward / partial credit design. Instead of assigning only binary solved/unsolved outcomes, each task is decomposed into a small number of semantically meaningful subtasks $\{s_1,\dots,s_K\}$. Each subtask has a normalized score $r_k \in [0,1]$, and the task reward is

$$R = \frac{\sum_{k=1}^K w_k\, r_k}{\sum_{k=1}^K w_k},$$

where $w_k \ge 0$ are subtask weights, usually equal unless the final objective is weighted more heavily (Li et al., 9 Jul 2026).

The paper describes three subtask types. Binary subtasks are programmatic pass/fail checks such as unit tests passing, a service running on the expected port, or a script completing without error. Continuous or thresholded subtasks are used for quantitative goals such as reproducing a figure, matching a metric within tolerance, obtaining a particular speedup, or matching a reference output on held-out examples. Episode-aggregating subtasks are used in campaign-style tasks such as games or repeated audits, where the score may be the fraction of successful episodes or mean normalized reward from the simulator.

A task is counted as solved if $R \ge \tau$. The paper uses a relaxed threshold such as $\tau = 0.95$ for main pass@1 reporting and also reports a strict threshold at $R \ge 1.0$ for perfect completion. Pass@1 is defined as

$$\text{pass@1}_{\tau} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[R_i \ge \tau],$$

with $N=46$ tasks (Li et al., 9 Jul 2026).

This grading scheme changes what benchmark scores mean. Because partial progress is measured directly, LHTB can distinguish between agents that fail immediately, agents that make substantial but incomplete progress, agents that stop too early, and agents that approach full completion but fail hidden verification.

4. Evaluation protocol and empirical results

The authors evaluate 15 frontier models under a shared terminal-agent harness: GPT-5.5, GPT-5.4, GPT-5.3 Codex, DeepSeek V4 Pro, Gemini 3.1 Pro, GLM 5.1, GLM 5.2, Kimi K2.6, Kimi K2.7 Code, MiniMax M3, Qwen3.7 Max, Qwen3.6 Plus, Doubao Seed 2.1 Pro, Hy3, and Grok 4.20. Most models are evaluated using the Harbor framework with the Terminus-2 agent harness; GPT-5.3 uses the Codex harness instead. An episode is one full task rollout in the terminal environment, ending when the agent reaches the end condition, exits early, or hits the 90-minute timeout (Li et al., 9 Jul 2026).

The benchmark is intentionally expensive and long-running. Across tasks and models, runs consume on average about 231 episodes per task, about 9.9M tokens per task, and about 85.3 minutes of execution time per task. The paper states that this is an order of magnitude harder than prior terminal benchmarks such as Terminal-Bench 2, which typically involve around 20–30 minutes and 20–30 episodes per task.

Performance is low even for the strongest tested models. GPT-5.5 is the strongest model, achieving 15.2% pass@1 at $R \ge 0.95$, corresponding to 7/46 tasks solved, and 10.9% at the strict perfect-reward threshold $r_k \in [0,1]$0. Across all 15 models, the mean pass rate is 4.3% at $r_k \in [0,1]$1 and 1.7% at $r_k \in [0,1]$2. The paper emphasizes that many models score partial credit without fully solving tasks, which is precisely why dense grading matters (Li et al., 9 Jul 2026).

These numbers indicate that frontier models can often make locally meaningful progress in long terminal workflows but still fail to complete the end-to-end process robustly. A plausible implication is that traditional binary terminal benchmarks can systematically underreport both near-misses and characteristic long-horizon breakdowns.

5. Failure modes and diagnostic interpretation

LHTB’s failure analysis centers on two dominant patterns. First, timeouts dominate: 79% of unresolved runs end because the agent runs out of the 90-minute budget while still working. These are not usually near-complete failures; mean reward on timed-out runs is only around 0.10 to 0.35 depending on the model. Second, about 19% of unresolved runs are early exits, where the agent stops on its own before the budget ends (Li et al., 9 Jul 2026).

Within the early-exit category, the paper identifies “false finishes”: runs in which the agent exits at a high reward but still fails the hidden verifier, indicating weak self-verification and poor stopping judgment. The authors identify 14 early exits with $r_k \in [0,1]$3 as false finishes. Examples include Kimi K2.7 Code stopping on duckdb-optimizer-closure at $r_k \in [0,1]$4, GLM 5.2 stopping on apex-ib244-matter at $r_k \in [0,1]$5, and several models stopping on apex-law433-matter in the $r_k \in [0,1]$6–$r_k \in [0,1]$7 range (Li et al., 9 Jul 2026).

The broader failure pattern is explicitly long-horizon-specific. Agents can do many locally correct things, yet still fail to convert partial progress into a fully verified final result before time expires, and they often misjudge completion. This distinguishes LHTB from shorter terminal benchmarks, where local command errors more often dominate. In LHTB, the benchmarked difficulty is not merely command issuance or localized patching, but the ability to sustain correct execution, accumulate verified progress, and terminate only when hidden checks are likely to pass.

6. Position within long-horizon agent evaluation

LHTB occupies the terminal-workflow end of a broader landscape of long-horizon benchmarks. LongDS treats long-horizon data analysis as persistent analytical state management over real notebook workflows, with 68 tasks, 2,225 turns, and pronounced degradation over long dependency spans; it is highly relevant to LHTB-style evaluation, but its primary object is notebook-state evolution rather than terminal execution (Xu et al., 28 May 2026). YC-Bench is a domain-grounded, partially observable, long-horizon terminal benchmark in which an agent runs a simulated startup over a one-year horizon and is judged by final company funds; it is conceptually aligned with LHTB but is centered on long-term planning and economic dynamics in a POMDP rather than dense reward decomposition of terminal workflows (He et al., 1 Apr 2026). SWE-Bench Pro evaluates long-horizon, enterprise-scale repository issue resolution with multi-file patches and human-augmented specifications; it complements LHTB by stress-testing long-horizon software engineering rather than general terminal work (Deng et al., 21 Sep 2025).

Subsequent work around related terminal benchmarks also clarifies what LHTB-like performance may depend on. CompactionRL argues that long-horizon terminal and coding agents are fundamentally limited by finite context windows and shows that context compaction can improve Terminal-Bench 2.0 Pass@1 substantially under compacted evaluation (Li et al., 6 Jul 2026). The proactive memory agent paper identifies “behavioral state decay” and shows that selective memory intervention improves Terminal-Bench 2.0 pass@1 for both weaker and stronger action agents (Wu et al., 9 Jul 2026). LiteCoder-Terminal, by contrast, focuses on synthesizing executable and verifiable terminal training environments at scale and reports improved performance on Terminal Bench 1.0, 2.0, and Pro after supervised fine-tuning and DMPO (Peng et al., 28 May 2026). These results do not evaluate LHTB directly, but they suggest that long-horizon terminal performance is constrained not only by raw reasoning quality, but also by context management, memory intervention, and the availability of verifiable training environments.

Within that broader ecosystem, LHTB’s distinctive contribution is the combination of terminal-only interaction, long wall-clock horizons, and dense reward-based grading. It asks not only whether an agent eventually finishes, but how reliably it progresses through long, open-ended terminal workflows and how often it mistakes partial progress for completion. The benchmark and evaluation harness are released, tasks are packaged in Harbor format, grading is deterministic and grounded in environment state, and the project page is https://zli12321.github.io/LHTB/ (Li et al., 9 Jul 2026).