ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking

Abstract: Reinforcement learning has substantially improved the performance of LLM agents on tasks with verifiable outcomes, but it still struggles on open-ended agent tasks with vast solution spaces (e.g., complex travel planning). Due to the absence of objective ground-truth for these tasks, current RL algorithms largely rely on reward models that assign scalar scores to individual responses. We contend that such pointwise scoring suffers from an inherent discrimination collapse: the reward model struggles to distinguish subtle advantages among different trajectories, resulting in scores within a group being compressed into a narrow range. Consequently, the effective reward signal becomes dominated by noise from the reward model, leading to optimization stagnation. To address this, we propose ArenaRL, a reinforcement learning paradigm that shifts from pointwise scalar scoring to intra-group relative ranking. ArenaRL introduces a process-aware pairwise evaluation mechanism, employing multi-level rubrics to assign fine-grained relative scores to trajectories. Additionally, we construct an intra-group adversarial arena and devise a tournament-based ranking scheme to obtain stable advantage signals. Empirical results confirm that the built seeded single-elimination scheme achieves nearly equivalent advantage estimation accuracy to full pairwise comparisons with O(N^2) complexity, while operating with only O(N) complexity, striking an optimal balance between efficiency and precision. Furthermore, to address the lack of full-cycle benchmarks for open-ended agents, we build Open-Travel and Open-DeepResearch, two high-quality benchmarks featuring a comprehensive pipeline covering SFT, RL training, and multi-dimensional evaluation. Extensive experiments show that ArenaRL substantially outperforms standard RL baselines, enabling LLM agents to generate more robust solutions for complex real-world tasks.

• The paper introduces tournament-based relative ranking to overcome the limitations of pointwise scalar rewards in open-ended RL tasks.
• It proposes multiple tournament topologies, with seeded single-elimination offering near-optimal ranking accuracy with linear computational cost.
• Empirical results show superior performance on benchmarks like Open-Travel and Open-DeepResearch, achieving significant win rate gains over conventional baselines.

ArenaRL: Tournament-Based Relative Ranking for Scalable RL in Open-Ended Agent Settings

Motivation and Problem Formulation

The application of reinforcement learning (RL) to open-ended LLM-based agents has been limited due to the inadequacy of ground-truth-based reward signals in unstructured, real-world tasks. Tasks such as travel planning or in-depth research do not admit a single gold-standard solution; their quality is determined across multi-dimensional, subjective axes. Mainstream RL approaches, even those leveraging LLM-as-judge frameworks, rely on pointwise scalar rewards assigned to individual trajectories. This work identifies and formalizes a critical pathology in these dynamics: discriminative collapse—as agent policies improve and generate high-quality but similar outputs, the pointwise scoring distribution within a group of trajectories collapses to a narrow range, with signal dominated by evaluator noise rather than meaningful differences in trajectory quality.

Figure 1: (a) Illustration of discriminative collapse: intra-group variance is suppressed and reward noise dominates; (b) ArenaRL performance: tournament-based ranking yields superior results over SFT, GRPO, and GSPO.

The authors propose reformulating the RL reward acquisition paradigm for open-ended agentic RL. Instead of unstable scalars, ArenaRL employs intra-group relative ranking as an optimization signal, using tournament-based pairwise comparison and process-aware multi-level rubrics for trajectory assessment.

ArenaRL Methodology

Process-Aware Pairwise Evaluation

ArenaRL introduces a process-aware LLM-as-judge $\mathcal{J}$, accepting as input the query, complete candidate trajectories (including intermediate chain-of-thought and tool invocations), and a rubric specifying detailed evaluation axes (e.g., logical coherence, tool effectiveness, answer fidelity). Critically, the evaluation proceeds in both directions (swapping generation order) to mitigate position bias—a well-documented artifact in LLM judgment schemes.

Tournament-Based Ranking Schemes

ArenaRL systematically explores five tournament topologies for efficient, robust intra-group ordering:

• Round-Robin—Exhaustive pairwise comparisons, providing an unbiased gold standard at $\mathcal{O}(N^2)$ cost.
• Anchor-Based Ranking—All exploratory samples compared only with a greedy-decoded anchor, achieving minimal $\mathcal{O}(N)$ cost but reduced fidelity among non-anchor samples.
• Seeded Single-Elimination—A novel hybrid: anchor-based seeding establishes a hierarchy, then a single-elimination binary tree is constructed, pairing high with low seeds, requiring $2N-2$ matches. This topology yields nearly optimal ranking accuracy against the round-robin baseline with substantially reduced computational overhead.

  Figure 2: ArenaRL pipeline replaces scalar pointwise rewards with intra-group ranking and employs multiple tournament topologies for optimal balance between sample efficiency and fidelity.

• Double-Elimination and Swiss-System—Incorporate losers’ brackets and dynamic rounds but were empirically outperformed by the seeded single-elimination.

In all variants, obtained ranks are mapped into normalized advantage signals for policy gradient optimization (cf. PPO-like schemes), with a KL-penalty to maintain reference model proximity.

Open-Ended Agent Benchmarks

ArenaRL introduces two new, full-cycle benchmarks:

• Open-Travel: Multi-constraint itinerary planning, tool-use, and reasoning challenges across five sub-tasks (direction, one-day, comparison, POI search, multi-day).
• Open-DeepResearch: Autonomous research tasks involving multi-turn search, retrieval, synthesis, and report generation.

  Figure 3: Construction process for Open-Travel and Open-DeepResearch benchmarks, spanning data curation, seed expansion, and quality assurance.

Benchmarks are constructed with end-to-end training pipelines, expert-annotated queries, and rigorous automated evaluation procedures, including multi-axis pairwise assessments by closed-source judge LLMs. Coverage extends beyond travel and research to standard open-ended writing and diverse real-world domains.

Empirical Analysis and Results

Extensive experiments evaluate ArenaRL and all reward/ranking baselines on the proposed benchmarks and three external open-ended writing datasets (WritingBench, HelloBench, LongBench-Write):

Tournament Selection: Seeded Single-Elimination achieves nearly round-robin-equivalent accuracy (average win rate 32.5% vs 32.9%) at only linear computational cost, outperforming all other sparse tournament topologies.

Open-Ended Agent Performance: On Open-Travel, ArenaRL yields an average win rate of 41.8%, compared to 16–17% for strong RL baselines (GRPO, GSPO) and outstripping all closed-source LLMs. In Open-DeepResearch, ArenaRL achieves 64.3% win rate with a 99% valid generation rate, addressing both effectiveness and robustness. Standard RL approaches stagnate or degrade owing to the inability to extract high-SNR optimization signals within the collapsed scalar reward regime.

Generalization to Writing Tasks: ArenaRL consistently outperforms both RL baselines and leading closed-source LLMs (GPT-4o, Claude-3.7-Sonnet) on open-ended generation, yielding a 6–7% absolute gain over GRPO/GSPO.

Scalability and Ablation: Increasing the group size $N$ during tournament sampling leads to monotonic improvements. Even minimal ($N=2$) pairwise comparison offers a regularizing effect beyond SFT, but the most substantial gains materialize for $N=16$.

Figure 4: (a) ArenaRL performance improves monotonically with group size $N$; (b) LLM and human evaluators exhibit 73.9% agreement, confirming metric reliability; (c) ArenaRL enables policy improvement even without SFT cold start, demonstrating resilience to absence of initial supervised data.

LLM-Judge Reliability: Agreement between LLM and human evaluation is high (73.9%), indicating that ArenaRL’s gains are not artifacts of overfitting judge preference, but reflect broad improvements.

Implications and Future Directions

Theoretical and Practical Significance

ArenaRL’s tournament-based reward collection reframes optimality in open-ended RL as a groupwise ranking problem, mitigating discriminative collapse—a pathology overlooked in prior RLHF and LLM-judging regimes. This method natively handles tasks with subjective, implicit, and multi-dimensional success criteria, expanding the tractable frontiers for RL-based LLM alignment and agent development.

Figure 5: Example Open-Travel prompt: showcases the type of complex, multi-constraint task processed by ArenaRL.

Robustness and Transferability

Extensive case analyses as well as deployment on Amap ecosystem data confirm that ArenaRL-trained models generalize and sustain strong performance both in deterministic (POI search) and genuinely open-ended (complex travel/research) scenarios. Planning, reasoning, and intent alignment capabilities improve substantially, correlating with business-relevant KPIs.

Future Prospects

Potential directions include extension of tournament-based reward modeling to multimodal agents, further scaling of tournament group sizes via distributed parallel computation, and automated rubric discovery for richer, domain-adaptive reward extraction.

Conclusion

ArenaRL establishes a scalable, robust paradigm for RL in open-ended agent environments by leveraging intra-group relative ranking via process-aware tournaments. The seeded single-elimination topology enables efficient, high-fidelity advantage estimation, breaking the limitations of scalar rewards in subjective, unstructured settings. This methodology enables systematic policy improvement, yielding agents with enhanced planning, reasoning, and generative capabilities across diverse, real-world benchmarks and applications (2601.06487).