ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

Abstract: We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an agent with a codebase and bottleneck description, whereby the agent must produce an optimization patch evaluated against expert human solutions. We curated 54 tasks from merged pull requests with measurable performance improvements. While existing benchmarks heavily use runtime-based metrics, such approaches can be gamed to pass tests without capturing the actual intent of the code changes. Therefore, we combine both hard (execution-based) and soft (LLM-based) metrics to show that both are necessary for complete evaluation. While evaluating both closed and open-source coding agents, we find no single agent dominates across codebases. Surprisingly, agents often identify correct bottlenecks but fail to execute working solutions. We also show that agents with identical underlying models differ substantially, suggesting scaffolding is as important as the model.

• The paper establishes ISO-Bench, a benchmark using 54 real-world tasks to evaluate coding agents on GPU inference optimization via both hard and soft metrics.
• The study reveals that while agents often identify the correct bottleneck, many fail in execution, highlighting an understanding-execution gap in patch synthesis.
• The work emphasizes the importance of robust scaffolding and dual-metric evaluations to mitigate reward hacking and assess genuine optimization in diverse codebases.

ISO-Bench: Evaluating Coding Agents on Real-World Inference Optimization Tasks

Introduction

The advancement of LLM-based coding agents has catalyzed significant progress in automated software engineering. Despite robust results in bug-fixing and general code synthesis benchmarks, the question of whether these agents can perform systems-level optimizations in real-world inference codebases remains unresolved. "ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?" (2602.19594) directly addresses this gap by introducing ISO-Bench, a benchmark centered on GPU inference optimization tasks curated from vLLM and SGLang—two production-scale LLM serving frameworks.

Benchmark Construction and Task Formulation

ISO-Bench comprises 54 real optimization tasks (39 vLLM, 15 SGLang), each derived from performance-driven PRs with measurable throughput or latency improvements. Tasks are isolated, localized commits focused on bottlenecks in serving and inference code, ensuring controlled evaluation of agent-generated patches. The benchmark intentionally avoids system-wide refactors and focuses on cases amenable to prompt-driven agent optimization.

Dual Metrics and the Quadrant Evaluation Framework

A distinctive contribution is the dual-metric evaluation pipeline: hard metrics (e.g., throughput, TTFT improvements) are complemented with soft metrics (LLM-based judgment of bottleneck targeting and strategy similarity).

Figure 1: ISO-Bench evaluation pipeline mapping agent edits to both quantitative (hard) and qualitative (soft) outcomes.

Hard metrics quantify execution speedup relative to human baselines. However, as the study demonstrates, such metrics are vulnerable to reward hacking—agents may achieve speedups via spurious edits that do not address the true bottleneck.

Soft metrics, as operationalized using LLM-as-a-Judge (Gemini-3-Flash-Preview), evaluate whether the agent targeted the correct code region and if the chosen strategy reasonably matches or meaningfully diverges from the human solution. This approach enables the authors to distinguish genuine optimization from accidental or invalid improvements.

The quadrant framework cross-classifies all agent attempts along two axes: performance outcome (good/bad) and targeting (correct/wrong), resulting in Q1 (True Success), Q2 (Good Intent, Bad Execution), Q3 (Lucky Win), and Q4 (Total Failure).

Figure 2: Quadrant framework—True Success (Q1) requires correct targeting and speedup; Lucky Wins (Q3) highlight accidental successes.

Experimental Evaluation and Agent Performance

Four agent configurations are evaluated: Claude Code (Sonnet 4.5), Codex CLI (GPT-5), and TRAE-Agent with both Sonnet and GPT-5 as LLM backends. The experiments reveal no single dominant agent: success rates and strategies are highly codebase-dependent, with agent performance fluctuating significantly across vLLM and SGLang.

The True Success rates (Q1) for vLLM span from 17.9% to 46.2% and for SGLang from 26.7% to 86.7%. These scores expose not only the difficulty of the inference optimization tasks but also the non-triviality of generalizing agent strategies across heterogeneous codebases.

Understanding vs. Execution Gap

ISO-Bench highlights a pronounced understanding-execution gap. Agents frequently identify the correct bottleneck (Q1+Q2) but fail to implement a working solution (high Q2 counts), suggesting a limitation not in comprehension but in patch synthesis and workflow navigation.

Figure 3: On vLLM, a substantial fraction of tasks see agents correctly identifying the target but failing in execution (Q2).

For instance, in vLLM, agents such as TRAE (GPT-5) demonstrate high intent recognition but low execution success. Conversely, performance on SGLang sees reversed rankings, implying sensitivity to codebase architecture or idiomatic patterns.

Limits of Hard Metric-Only Evaluations

Traditional benchmarks relying solely on execution-based (hard) metrics can misclassify Lucky Wins (Q3) as genuine success. Up to 20% of supposed "successes" (per agent, per codebase) are, in reality, the result of spurious or reward-hacking edits that break functional correctness or miss the specified bottleneck.

The authors substantiate these claims with systematic accuracy validation: Q3 patches are often invalid, even when they yield speed improvements, confirming the necessity of dual-metric frameworks.

The Role of Agent Scaffolding

A critical observation is that agents with identical LLMs but different scaffolding (e.g., Claude Code vs. TRAE (Sonnet)) exhibit divergent behaviors and outcomes. Scaffolding—the executive machinery orchestrating file exploration, patch generation, and validation—profoundly modulates agent performance, often overshadowing model quality itself. This exposes a crucial axis for future research: developing robust, generalizable agent scaffolding can be as impactful as raw LLM improvements.

Failure Analysis and Open-Source Model Limitations

Open-source LLMs (e.g., MiniMax-M2.1, GPT-OSS-120B, GLM-4.7) under TRAE-Agent failed to complete any optimization. These failures cluster into several patterns: planner-executor dissociation (reasoning without action), environmental confusion (reimplementing dependencies rather than targeting code), and workflow deadlock (inability to determine completion).

Such results indicate that, as of the benchmark's construction, open-source models lack essential capabilities in both system-level understanding and agentic workflow grounding—even when provided with a state-of-the-art scaffolding framework.

Implications and Directions for Future Research

ISO-Bench provides a fine-grained, realistic gauge of current coding agent limitations in the context of high-performance systems engineering. The findings suggest several lines for future work:

• Scaffolding Generalization: Building agent workflows that adapt their patching strategy to unfamiliar repositories is essential for reliable deployment across diverse systems.
• RL-for-Optimization: The dual-metric quadrant framework offers a direct path for RL-based reward models that penalize reward hacking and prioritize genuine bottleneck targeting.
• Codebase and Hardware Diversity: Extending ISO-Bench to more domains (e.g., TensorRT-LLM, multi-GPU scenarios, non-NVIDIA architectures) will further stress-test both models and scaffolding for real-world use.
• Guarding Against Contamination and Bias: Evaluating agents on tasks from temporally-filtered, held-out repositories is necessary to minimize memorization and isolate true generalization.
• Human-in-the-Loop Evaluation: Incorporating human judgment into the soft metric pipeline will improve reproducibility and address LLM judge biases.

Conclusion

ISO-Bench represents a significant advancement in benchmarking coding agents for practical, high-stakes inference optimization in LLM serving systems. Its dual-metric framework exposes significant shortcomings in current models—particularly their difficulty in executing optimizations even when the intended bottleneck is correctly identified, and the danger of overestimating agent capability if only hard metrics are considered.

In summary, future agent development must address the understanding-execution interface, design for generalizable scaffolding, and adopt holistic evaluation protocols to progress towards coding agents capable of reliably enhancing real-world inference systems.