Task Bench: A Task-Centric Benchmark

Updated 15 July 2026

Task Bench is a scalable benchmark that models complex tasks as explicit dependency graphs to evaluate parallel runtimes and automation systems.
It standardizes comparisons across systems by employing configurable parameters like METG and diverse dependency motifs for precise performance analysis.
The framework has evolved to inspire benchmarks for LLM automation, workspace reasoning, and embodied planning, addressing real-world operational challenges.

In the literature, “Task Bench” denotes both a specific parameterized benchmark for parallel runtime performance and a broader naming pattern for benchmark suites that operationalize complex work as structured tasks with explicit dependencies, controllable environments, and automated evaluation. The original “Task Bench: A Parameterized Benchmark for Evaluating Parallel Runtime Performance” distilled applications into task graphs for apples-to-apples comparisons across parallel and distributed systems, while later benchmarks such as “TaskBench,” “TPS-Bench,” “Workspace-Bench,” “Terminal-Bench,” “WorkBench,” “LoTa-Bench,” “ET-Plan-Bench,” and “ToG-Bench” adapted the task-centric paradigm to LLM-based automation, tool orchestration, workspace reasoning, command-line execution, embodied planning, and task-oriented grounding (Slaughter et al., 2019).

1. Origins in parameterized runtime benchmarking

The original Task Bench was introduced as a parameterized benchmark that models applications as task graphs in a two-dimensional iteration space with time along the vertical axis and parallel tasks along the horizontal axis. Its central design goal was orthogonality between benchmark specification and system-specific implementation: every benchmark constructed with Task Bench runs unmodified on every supported runtime, reducing study effort from $O(mn)$ to $O(m+n)$ for $m$ benchmark scenarios and $n$ systems. The framework exposes configurable dimensions such as height, width, dependence motif, radix, kernel type, iterations, span, scratch, imbalance, output payload, number of concurrent graphs, and scheduling or placement effects (Slaughter et al., 2019).

Task Bench’s dependence motifs include trivial, stencil, sweep, FFT-like butterfly, tree, and random graphs, allowing it to emulate halo exchanges, hierarchical reductions, spectral transforms, and irregular dependencies. The benchmark introduced Minimum Effective Task Granularity, or METG, defined as the smallest average task duration at which an application still achieves a target efficiency threshold, with METG(50%) used throughout the study. A major empirical result was that, when running at scale, $100\ \mu s$ is the smallest granularity that even the most efficient systems can reliably support with current technologies (Slaughter et al., 2019).

Later systems work reused Task Bench as a common yardstick for runtime comparisons. “Quantifying Overheads in Charm++ and HPX using Task Bench” measured Charm++, HPX, MPI, OpenMP, and MPI+OpenMP under stencil workloads, overdecomposition, and multi-node scaling, reporting, for example, single-task-per-core METG values of $3.92963\ \mu s$ for MPI, $9.7605\ \mu s$ for Charm++, $19.3313\ \mu s$ for HPX distributed, and $22.3684\ \mu s$ for HPX local (Wu et al., 2022). “Exploring Performance-Productivity Trade-offs in AMT Runtimes: A Task Bench Study of Itoyori, ItoyoriFBC, HPX, and MPI” extended this use to application efficiency, METG, Lines of Code, and Number of Library Constructs, showing that MPI achieves the highest efficiency for regular, communication-light workloads, HPX maintains stable efficiency under load imbalance across varying node counts, and Itoyori leads the study’s programmer-productivity metrics (Lahnor et al., 21 Jan 2026).

2. Dependency formalisms and benchmark semantics

A defining feature across task-centric benchmarks is the explicit representation of dependencies. In the original Task Bench, dependencies are encoded by analytical task-graph motifs that determine which tasks in timestep $t$ depend on outputs from timestep $O(m+n)$ 0 (Slaughter et al., 2019). In “TaskBench: Benchmarking LLMs for Task Automation,” the analogous abstraction is the Tool Graph, a directed graph $O(m+n)$ 1 in which nodes are tools and edges encode resource or temporal dependencies; subgraphs are sampled in node, chain, or DAG modes to control orchestration complexity (Shen et al., 2023).

Subsequent agent benchmarks specialized the same dependency idea around different artifacts. TPS-Bench decomposes each task into subtasks $O(m+n)$ 2 with directed edges $O(m+n)$ 3 encoding prerequisites, maps each node to a “basic tool” interface in an MCP repository, and treats planning and scheduling as a joint decision problem over a subtask DAG, ready sets, and sequential versus parallel execution (Xu et al., 3 Nov 2025). Workspace-Bench defines a file dependency graph $O(m+n)$ 4 with typed edges for read/write, semantic reference, and lineage, then packages each task as a tuple $O(m+n)$ 5 containing an instruction, essential files, rubrics, and a task-specific dependency graph (Tang et al., 5 May 2026). C $O(m+n)$ 6-Bench formalizes multi-task agenting through a specialized process model $O(m+n)$ 7 and evaluates correctness against enumerated serial, parallel, and serial+parallel tool-dependency paths (Yu et al., 24 May 2025).

This suggests a broad convergence toward graph-structured benchmark design. The concrete node and edge semantics differ—tasks, tools, files, or policies—but the underlying methodology is similar: benchmark difficulty is controlled by dependency density, branching structure, hidden prerequisites, and the need to preserve valid execution order.

3. Tool-use and workflow automation benchmarks

Task-centric agent benchmarks differ chiefly in the environment over which dependencies are instantiated.

Benchmark	Environment	Main evaluation focus
TaskBench	Tool graphs in Hugging Face, multimedia, and daily life APIs	decomposition, tool invocation, parameter prediction
WorkBench	five databases and 26 workplace tools	outcome-centric workplace task execution
TPS-Bench	15 MCP servers exposing 141 tools	tool planning and scheduling
C $O(m+n)$ 8-Bench	400 tools grouped into 50 categories, totaling 1600 tools	robustness under tool dependencies, hidden information, and policy switching

TaskBench introduced a three-stage evaluation stack—task decomposition, tool invocation, and parameter prediction—grounded in Tool Graphs and a back-instruct data engine. Its final human-verified datasets contain 7,546 Hugging Face samples, 5,584 multimedia samples, and 4,320 daily-life samples, and TaskEval reports Rouge and BERTScore for step descriptions, node and edge F1 for graph structure, Normalized Edit Distance for chains, and exact-match F1 variants for parameter names and values. The benchmark reported strong ranking agreement with human judgments, with average Kendall’s $O(m+n)$ 9 and Spearman’s $m$ 0, and identified edge prediction and parameter values as persistent bottlenecks even for GPT-4 (Shen et al., 2023).

WorkBench moved the task-benchmark idea into a realistic workplace sandbox comprising five simulated databases, 26 tools, and 690 tasks. Its key methodological contribution was outcome-centric evaluation: agents are judged by whether their actions produce the unique, unambiguous target state in the sandbox rather than by similarity to a reference action trace. In the reported ReAct-style baselines, GPT-4 achieved 49% accuracy when given only the required tools and 43% when given all tools, while Llama2-70B achieved 3% and 0%, respectively. GPT-4’s accuracy fell from 75% on 0-action tasks to 36% on 1+ action tasks and 18% on 2+ action tasks, illustrating the rapid difficulty increase induced by multi-step execution (Styles et al., 2024).

TPS-Bench targets compounding tasks that require both tool planning and tool scheduling over a large MCP ecosystem. It aggregates 15 MCP servers exposing 141 tools and collects 200 compound tasks in two tiers: TPS-Bench-Easy with at most five subtasks and TPS-Bench-Hard with up to 50 subtasks. Evaluation jointly emphasizes Task Completion Rate, Tool Selection Score, end-to-end execution time, token usage, number of tool-call turns, and optional cost-of-pass. The reported results show that planning is generally “reasonable,” but scheduling strategies diverge sharply: on Hard tasks, GLM-4.5 achieved 64.72% completion with 35.0 turns and 217.8 seconds per task, whereas GPT-4o used roughly 2–3 turns and 76.84 seconds but reached only 45.08% completion. A reinforcement-learning study using GRPO on Qwen3-1.7B improved Hard completion from 26.75% to 33.13% while reducing time from 42.0 seconds to 36.1 seconds (Xu et al., 3 Nov 2025).

C $m$ 1-Bench focuses less on nominal task success than on the factors that disturb agent robustness in multi-tasking. It contains 256 single-task items and 768 multi-task items, evaluates 49 mainstream agents, and isolates three stressors: complex tool relationships, critical hidden information, and dynamic policy-type switching. Its fine-grained metrics include Accomplish Progress, Optimal Path Rate, Volatility Factor, Debiased Descent Direction, and Policy Transition Frequency. The best overall accuracy under the multi-tasking protocol was 55.34%; the study reports that inter-tool dependencies cause a 28.7% average accuracy decline, Long-Context dependencies induce an additional 11.0% average decline versus Omit and Reference, and each additional task causes a 10.1% average drop (Yu et al., 24 May 2025).

4. Embodied planning and task-oriented grounding

A second major branch of task-centric evaluation concerns embodied agents. LoTa-Bench was introduced as a language-oriented Task Bench for high-level planning in home-service environments, standardizing natural-language instructions, skill vocabularies, and automatic success checking across ALFRED with AI2-THOR and WAH-NL with VirtualHome. The planner objective is to select the next high-level skill maximizing the probability of completing the instruction given the instruction and skill history, and the benchmark systematically studies model family, prompt construction, in-context example selection, feedback-based replanning, and fine-tuning. GPT-4 reached 40.38% success on ALFRED under chat prompting, compared with 21.36% for GPT-3, while on WAH-NL GPT-3 achieved 40.82% average subgoal success and GPT-4 achieved 34.17% (Choi et al., 2024).

ET-Plan-Bench sharpened the embodied-planning diagnosis around spatial cognition and temporal or causal understanding. It uses VirtualHome 3.0 and Habitat 2.0 as backends, generates tasks with relation constraints, occlusion sources, and action-order dependencies, and measures Success Rate, Sequence Length, Longest Common Subsequence, LCS Ratio, and Moving Distance. The benchmark reports that navigation with layout priors is substantially easier than unconstrained navigation, and that spatial relations and temporal dependencies remain difficult: GPT-4 achieved 90.77% success on “Navi + Layout Map,” 62.61% on “Navi + Relation,” 54.81% on “Relation + Occlusion_Distance,” and 58.60% on temporal dependency tasks (Zhang et al., 2024).

ToG-Bench extended the task-benchmark idea from action planning to perception aligned with embodied intent. It is described as the first task-oriented spatio-temporal video grounding benchmark for egocentric videos, built from 100 ScanNet-derived clips, 2,704 task-oriented instructions, and 4,194 annotated object instances spanning 177 functional object categories. Its key distinctions are task-oriented grounding, explicit-implicit dual grounding, and one-to-many grounding, with decoupled evaluation for semantic recognition and spatio-temporal grounding at both object and task levels. GPT-5 obtained T-Acc 89.42%, T-m_tIoU 36.64%, and T-m_vIoU 29.88%, while Gemini 2.5 Pro obtained T-Acc 80.14%, T-m_tIoU 41.58%, and T-m_vIoU 38.48%; both models showed marked degradation from explicit to implicit grounding and from single-object to 3+ object tasks (Xu et al., 3 Dec 2025).

These benchmarks collectively shift “task” from a static instruction to an interactive program over skills, relations, affordances, and state predicates. This suggests that, in embodied settings, task-centric evaluation increasingly measures not only whether a plan reaches a goal but whether the model can preserve causal order, recover from feedback, and ground functionally relevant objects under partial observability.

5. Workspace-scale and terminal-scale task environments

Workspace-Bench and Terminal-Bench generalize task-centric evaluation to two environments central to contemporary digital work: heterogeneous file ecosystems and command-line interfaces.

Workspace-Bench defines Workspace Learning as the capability to identify, reason over, exploit, and update dependencies among heterogeneous files in a worker’s workspace. The benchmark constructs five worker profiles, 74 file types, and 20,476 files, with up to 11,020 files in a single workspace and average directory depth up to eight levels. It curates 388 tasks and 7,399 total rubrics, with each task carrying its own dependency graph and evaluation split across result-oriented, foundation, and process-oriented checks. On Workspace-Bench-Lite, a 100-task subset that reduces evaluation cost by about 70%, the average agent performance was 47.4% Rubric Pass Rate, the best agent reached 68.7%, and the human-in-the-loop baseline reached 80.7% (Tang et al., 5 May 2026).

Terminal-Bench 2.0 evaluates long-horizon, outcome-driven agent work in Dockerized command-line environments. Its curated release contains 89 tasks selected from 229 submissions by 93 contributors, each defined by a containerized environment, an instruction, a set of tests, a human-written oracle solution script, and a time limit. Verification is based strictly on the final container state rather than command traces. The benchmark reports that frontier models and agents score less than 65%, with the best listed result being Codex CLI + GPT-5.2 at 62.9% ± 3.0%, followed by Terminus 2 + Claude Opus 4.5 at 57.8% ± 2.5% and Terminus 2 + Gemini 3 Pro at 56.9% ± 2.5%. The study ran 32,155 trials and found that some tasks remained unsolved by any model or agent (Merrill et al., 17 Jan 2026).

The contrast between these two benchmarks is instructive. Workspace-Bench centers file dependency graphs, lineage tracing, and cross-file reasoning; Terminal-Bench centers real command execution, system tooling, and end-state verification. A plausible implication is that “Task Bench” has expanded from abstract task-graph scheduling into benchmarks of operational work itself, where the task is inseparable from the substrate—filesystem, shell, services, artifacts, or container state—on which it is executed.

6. Evaluation methodologies, metrics, and recurrent failure modes

Task-centric benchmarks are distinguished as much by their metrics as by their environments. Runtime-oriented Task Bench emphasizes efficiency curves and METG under controlled task-graph parameters (Slaughter et al., 2019). TaskBench for LLM automation uses Rouge-1, Rouge-2, BERTScore F1, node and edge F1, Normalized Edit Distance, and parameter F1 variants (Shen et al., 2023). TPS-Bench adds end-to-end execution time, token usage, tool-call turns, and cost-of-pass to completion-based evaluation (Xu et al., 3 Nov 2025). Workspace-Bench uses Rubric Pass Rate, TCR@p, and node or edge F1 for dependency graph recognition (Tang et al., 5 May 2026). Terminal-Bench reports resolution rate with 95% confidence intervals (Merrill et al., 17 Jan 2026). ToG-Bench separates recognition accuracy from temporal and spatial grounding, including m_tIoU, R1@ $m$ 2, m_vIoU, AP@ $m$ 3, and strict task-level variants (Xu et al., 3 Dec 2025). C $m$ 4-Bench adds robustness-sensitive metrics such as Volatility Factor and Debiased Descent Direction (Yu et al., 24 May 2025).

A second recurring theme is automated judging with increasingly explicit ground truth. WorkBench and Terminal-Bench avoid LLM judging by scoring exact environment state changes or end-state tests (Styles et al., 2024). TaskBench anchors evaluation to sampled Tool Graphs and exact parameter triples (Shen et al., 2023). TPS-Bench uses Gemini-2.5-Flash as an LLM-as-a-judge and reports Pearson 0.8375 for completion scores and 0.7590 for subtask counts against humans (Xu et al., 3 Nov 2025). Workspace-Bench adopts an agent-as-a-judge but pairs it with binary rubrics, execution traces, and task-specific dependency graphs (Tang et al., 5 May 2026). This suggests an unresolved methodological split between benchmarks that privilege executable state comparison and benchmarks that rely on judged decomposition or completion because outputs are open-ended.

Failure analyses are similarly consistent across domains. TPS-Bench identifies over-parallelization, over-sequentialization, tool selection inefficiency, context overflow, and judge sensitivity (Xu et al., 3 Nov 2025). WorkBench records parameter grounding failures such as hallucinated recipients, invalid JSON actions, and harmful side effects like emailing the wrong person (Styles et al., 2024). Workspace-Bench highlights heterogeneous file understanding and lineage tracing as notable bottlenecks (Tang et al., 5 May 2026). Terminal-Bench groups failures into execution, coherence, and verification classes, and reports common command-level errors such as “Command not found on PATH” and “Failures when running executables” (Merrill et al., 17 Jan 2026). LoTa-Bench finds that action planning failures and object selection failures account for more than 60% of GPT-3’s ALFRED errors (Choi et al., 2024). ET-Plan-Bench reports deterioration under occlusion, relation verification, and strict ordering (Zhang et al., 2024). ToG-Bench emphasizes implicit reasoning failures, multi-object coordination, and degradation on long videos (Xu et al., 3 Dec 2025).

7. Open problems and research directions

The benchmark family leaves several questions unresolved. Task Bench in high-performance computing identifies runtime overhead, communication topology, and latency hiding as central issues, but later runtime studies note that productivity metrics such as Lines of Code and Number of Library Constructs remain only indicative because counting methodology is not always fully specified (Lahnor et al., 21 Jan 2026). In agent benchmarks, tool coverage and environment fidelity remain limited: TPS-Bench notes that 141 MCP tools cover many domains but not all realistic APIs, Workspace-Bench notes that five personas do not encompass all industries or collaboration modes, and WorkBench acknowledges that real inboxes and CRMs are larger and messier than the sandbox (Xu et al., 3 Nov 2025).

Evaluation fidelity is another major open problem. Workspace-Bench and TPS-Bench both rely partly on automated judging and explicitly call for continued human auditing or rule-based checks (Tang et al., 5 May 2026). Terminal-Bench flags risks from internet access, training contamination, and rapid benchmark saturation, with plans for new task sets (Merrill et al., 17 Jan 2026). ET-Plan-Bench notes a sim-to-real gap, imperfect perception modules, and missing left/right semantics (Zhang et al., 2024). ToG-Bench identifies limited scene diversity, simulated egocentric trajectories, and 1 fps box-level tracking as current constraints (Xu et al., 3 Dec 2025). C $m$ 5-Bench points to data-scaling limits imposed by expert-in-the-loop annotation and exhaustive policy-transition coverage (Yu et al., 24 May 2025).

Taken together, these limitations suggest that “Task Bench” has become less a single benchmark than a research program. Across runtimes, tool-using agents, workspaces, terminals, embodied simulators, and egocentric video, the common objective is to measure task execution under explicit dependency structure rather than isolated prompt response. The central research challenge is no longer merely whether a model can produce a plausible answer, but whether it can preserve correctness, efficiency, robustness, and process fidelity when tasks are long-horizon, stateful, dependency-rich, and operationally consequential.