PyVision-RL: Forging Open Agentic Vision Models via RL

Abstract: Reinforcement learning for agentic multimodal models often suffers from interaction collapse, where models learn to reduce tool usage and multi-turn reasoning, limiting the benefits of agentic behavior. We introduce PyVision-RL, a reinforcement learning framework for open-weight multimodal models that stabilizes training and sustains interaction. Our approach combines an oversampling-filtering-ranking rollout strategy with an accumulative tool reward to prevent collapse and encourage multi-turn tool use. Using a unified training pipeline, we develop PyVision-Image and PyVision-Video for image and video understanding. For video reasoning, PyVision-Video employs on-demand context construction, selectively sampling task-relevant frames during reasoning to significantly reduce visual token usage. Experiments show strong performance and improved efficiency, demonstrating that sustained interaction and on-demand visual processing are critical for scalable multimodal agents.

• The paper introduces a curriculum-aware, reward-incentivized RL framework that mitigates interaction collapse and promotes sustained multimodal reasoning.
• It employs dynamic Python tooling for selective image and video processing, thereby enhancing token efficiency and tool utilization.
• Empirical results demonstrate state-of-the-art improvements across visual search, multimodal, and mathematical reasoning tasks over traditional baselines.

PyVision-RL: An RL Framework for Open Agentic Vision Models

Introduction and Motivation

PyVision-RL ("-RL") establishes a formalized reinforcement learning (RL) framework designed for open-weight multimodal models with strong agentic capabilities for both image and video understanding (2602.20739). The paper directly addresses persistent failure modes encountered in agentic multimodal model RL—primarily, interaction collapse, where models regress to minimal tool use and truncated reasoning chains, consequently failing to leverage the potential of multi-turn interaction. To counteract this, the authors introduce a unified pipeline for both image and video reasoning, denoted as -Image and -Video, incorporating dynamic Python-based tooling, a robust RL objective, and a novel rollout generation protocol.

Figure 1: Agentic scaffolds of -RL enabling unified agent-environment interaction and dynamic tooling for image and video tasks.

Agentic Scaffold and Dynamic Tooling

The presented agentic scaffold allows the model to interleave language-based reasoning with executable Python code snippets, supporting a closed-loop where code output and rendered multimedia artifacts are injected recursively into model context. For image understanding (-Image), images are injected into both the model’s context and the execution environment; for video understanding (-Video), only the system prompt is injected into the context, with the complete video managed in the Python runtime.

Crucially, -Video introduces on-demand context construction, departing from uniform frame sampling. Instead, frame selection and rendering are delegated to code generated by the agent itself, which samples, processes, and injects only relevant frames as required for the downstream spatial or reasoning task.

Figure 2: Comparison between uniform frame sampling and PyVision-RL’s on-demand context construction for video understanding.

This shift tightly controls context length and visual token consumption without sacrificing essential perceptual information—a recurrent bottleneck in scaling multimodal LLMs to long videos.

RL Formulation, Rollout Generation, and Optimization

Accumulative Tool Reward

To maintain sustained interaction and avoid the collapse into one-step solutions, the RL objective introduces an accumulative tool reward:

$$R = R_\mathrm{acc} + 0.1 \cdot n_\mathrm{tc} \cdot \mathbf{1}_{\{R_\mathrm{acc} = 1\}}$$

where $R_\mathrm{acc}$ is the accuracy-based reward, and $n_\mathrm{tc}$ is the number of tool calls (included only when the trajectory is correct). This mechanism creates a direct incentive for multi-step tool usage and discourages the degenerate minimization of tool invocation.

Oversampling–Filtering–Ranking Rollouts

Training stability and gradient informativeness are further enforced through an oversampling-filtering-ranking procedure for rollout selection. Rollouts are first oversampled, invalid groups (with zero reward variance or execution failures) are filtered out, and the remaining candidates are prioritized by intra-group reward standard deviation, selecting those with maximal learning signal and moderate difficulty.

Figure 3: Overview of the Oversampling-Filtering-Ranking Framework for Rollout Generation, stabilizing RL by focusing updates on informative, diverse rollouts.

Standard deviation normalization is removed from advantage computation (contrary to original GRPO), empirically shown to reduce gradient variance and eliminate suppression of correct but concise tool-using trajectories.

Empirical Results

Image Reasoning Results

On a battery of visual search, multimodal reasoning, and agentic reasoning tasks, -Image achieves consistent state-of-the-art results. Notably, it outperforms the base Qwen2.5-VL-7B by +10.2% (V*), +6.5% (HRBench-4K), and +6.4% (HRBench-8K) in visual search, and surpasses competing agentic models like DeepEyes-v2 by +9.6% on WeMath. Performance gains are especially evident on agentic reasoning benchmarks requiring long-horizon reasoning chains, demonstrating the benefit of explicitly aligning the reward structure with sustained tool use.

Figure 4: RL training dynamics for -Image show stable convergence, increasing tool use, and monotonic validation gains.

Video Reasoning and Efficiency

-Video demonstrates a superior accuracy–efficiency trade-off on VSI-Bench: with only 5K visual tokens per sample, it achieves 44.0% accuracy, compared to the 38.0% at 45K tokens for Qwen2.5-VL-7B—a nearly order-of-magnitude token reduction. Unlike prior work that relies on static, uniform sampling, the agent's autonomous frame selection ensures that token budget is allocated to exclusive, task-relevant evidence.

Figure 5: Accuracy versus token usage for -Video and baselines demonstrates efficiency advantages of on-demand context construction.

This strategy generalizes across multiple task archetypes; case studies illustrate how the agent leverages programmatic frame access for metric estimation and object counting, producing verifiable, interpretable intermediate outputs.

Ablation and Analysis

Ablation studies underscore that removing the accumulative tool reward or the standard deviation-based rollout sorting significantly degrades long-term performance and tool use. Increasing the maximum turn budget yields a higher performance ceiling in late-stage training.

Figure 6: Ablation of RL pipeline components, indicating the necessity of each for optimal performance.

Analysis of sample advantages reveals that standard deviation sorting not only enhances curriculum by prioritizing “edge-of-competence” samples, but also mitigates adverse gradient signals from correct samples with relatively fewer tool calls.

Figure 7: Ratio of positive samples with negative advantage plummets when standard deviation sorting is applied, stabilizing optimization.

Practical and Theoretical Implications

By unifying agentic reasoning across image and video and integrating dynamic Python tooling, PyVision-RL presents an open, extensible baseline for complex multimodal interaction. The framework’s rollout generation and reward composition strategies provide a principled template for future multimodal RL pipelines, especially for scenarios where excessive context length and degenerate RL convergence are limiting factors.

Practically, PyVision-RL’s explicit, interpretable tool use and execution traces facilitate model auditing and correction, in addition to delivering measurable improvements in performance and token efficiency. The theoretical contribution centers around resolving the incentive mismatch—previously unaddressed for multimodal agents—between single-turn supervision and multi-turn, multi-tool RL.

Speculation on Future Work

PyVision-RL’s extensible agentic scaffold invites the integration of additional modalities (e.g., audio, 3D environments) and more sophisticated external toolchains (beyond PIL/matplotlib/NumPy). The data-centric observation that curriculum learning via rollout diversity and difficulty sorting enhances generalization is likely to generalize to larger and more heterogeneous interactive agents. Further, the open-weight nature unlocks research in safe deployment, reflectivity in agent-environment interaction, and robust evaluation protocols across long-horizon, open-ended tasks.

Conclusion

PyVision-RL provides a rigorous, open-source agentic RL framework capable of training scalable, context-efficient, multi-turn multimodal models for both image and video reasoning (2602.20739). Its main contributions—dynamic Python tool integration, accumulative tool reward, and robust rollout selection—collectively avoid interaction collapse, stabilize RL, and yield state-of-the-art empirical results while advancing theoretical understanding of multi-agent training dynamics and curriculum design in agentic vision models.