BRPO: Balanced Reflective Policy Optimization

• BRPO is a reinforcement learning framework that integrates variance-reduced credit assignment, behavior regularization, and composite reward schemes to improve stability and sample efficiency across domains.
• It enables anytime reasoning in LLMs, balanced visual reflection in VLRMs, and symmetric regularization in offline RL through tailored strategies and baseline innovations.
• Empirical evaluations demonstrate BRPO's superior performance over existing methods in token-efficient reasoning, visual question answering accuracy, and robust offline control.

Balanced Reflective Policy Optimization (BRPO) is a reinforcement learning framework that generalizes behavior regularization, policy optimization, and reflection-guided reasoning in contemporary LLMs, vision-language reasoning models (VLRMs), and offline RL. BRPO encompasses distinct instantiations in different problem domains, unified by their use of variance-reduced credit assignment, behavioral or reflective regularization, and multi-objective reward schemes to enhance sample efficiency, stability, and robustness under distributional constraints, budgeted inference, and reflection-driven reasoning.

1. Conceptual Overview

BRPO methods are derived to address limitations of prior policy optimization approaches, such as Group Relative Policy Optimization (GRPO), which either optimize only final performance under fixed computation budgets or rely exclusively on asymmetric divergence measures. BRPO encompasses:

• Budget Relative Policy Optimization—Variance-reduced RL for anytime reasoning in LLMs, optimizing over variable token budgets and dense verifiable rewards (Qi et al., 19 May 2025).
• Balanced Reflective Policy Optimization in VLRMs—Rule-based RL to autonomously regulate when and how to perform visual-text reflections during multimodal reasoning, balancing reflection frequency and length to mitigate hallucinations (Chu et al., 29 May 2025).
• Behavior-Regularized Policy Optimization in Offline RL—Symmetric divergence-based regularized RL, enabling practical actor-critic learning without directional policy bias, using finite Taylor expansions for closed-form policy solutions (Zhu et al., 6 Aug 2025).

Across these domains, BRPO is characterized by its tailored variance-reduction strategies, composite reward design, and enhanced policy regularization mechanisms.

2. Formal Problem Instantiations

BRPO is realized in multiple decision-making settings:

A. Anytime Reasoning in LLMs

The framework models reasoning as an MDP, where at each step a token is generated, and the process is truncated at a stochastic budget $b$. The policy $\pi_\theta$ produces chain-of-thought tokens $z = (z_1, z_2, ...)$; a summary policy $\pi_\phi$ generates the answer $y$. The objective maximizes the expected reward across sampled budgets $b \sim p_\mathcal{B}$, formalized as

$$J_{\mathrm{anytime}}(\theta, \phi) = \mathbb{E}_{b, x, z} \left[ r_\phi(x, z_{\leq b}) \right]$$

where $r_\phi(x,z_{\leq b})$ is the expected verifiable reward of the summary given truncated reasoning. Budget weighting via distribution $p_\mathcal{B} = \{P_j\}$ enables robust performance across computational constraints and facilitates token efficiency (Qi et al., 19 May 2025).

B. Reflective Reasoning in VLRMs

BRPO in Qwen-LookAgain combines the PPO-style objective with a composite reward $$r_i = r^{\mathrm{format}}_i + r^{\mathrm{acc}}_i + r^{\mathrm{bal}}_i$$, where:

• $r^{\mathrm{format}}_i \in \{0,1\}$ ensures segment structuring and reflection placement;
• $r^{\mathrm{acc}}_i \in \{0,1\}$ verifies correct final answers;
• $r^{\mathrm{bal}}_i \in [0,1]$ penalizes deviation from ideal reflection length $\lambda$.

The model's policy learns to emit reflection tokens, balancing the number $N_r$ and average length $\overline{L}_r$ of reflection blocks. The KL-regularized clipped-reward objective is optimized per-group, normalized by intra-group statistics, preventing mode collapse and catastrophic forgetting (Chu et al., 29 May 2025).

C. Symmetric Regularization in Offline RL

BRPO instantiates behavior regularization via symmetric $f$-divergences (e.g., Jensen–Shannon, Jeffreys), regularizing the policy $\pi$ against a fixed behavior policy $\mu$:

$$\pi_{\mathrm{Reg}} = \arg\max_{\pi} \mathbb{E}_{s,a}[Q(s,a)] - \tau D_{f}^{\mathrm{sym}}(\pi(\cdot|s)\|\mu(\cdot|s))$$

Finite-order Taylor expansion ($N=2$) around $z=1$ yields analytic sparse solutions:

$$\pi_{\mathrm{Reg}}(a|s) \propto \mu(a|s) \left[1 + \frac{Q(s,a) - \alpha(s)}{\tau'} \right]_+$$

Loss minimization decomposes the $f$-divergence into asymmetric and conditional symmetry terms, the latter approximated via Taylor series, enabling stable optimization (Zhu et al., 6 Aug 2025).

3. Algorithmic Details and Pseudocode

AnytimeReasoner with BRPO

for each policy-gradient iteration: Sample a batch of questions {x_i} for each x_i: for g in 1..G: Sample full chain-of-thought z^{(i,g)} ~ π_θ up to max budget for each budget b_j: Truncate z^{(i,g)}_{≤b_j} Sample summaries y^{(i,g,k)} ~ π_φ Compute and aggregate summary reward r_φ^{(i,g,j)} for token time t in z^{(i,g)}: Compute budget index j_t, cumulative reward R, baselines V₁/V₂, interpolation V Advantage: A_t = R - V Update π_θ via PPO with {A_t} Update π_φ for summary with decoupled objective

(Qi et al., 19 May 2025)

BRPO in VLRMs (Qwen-LookAgain)

for each iteration: Sample B questions {q^b} for each q^b: Generate G candidate outputs {o^b_i} For each o^b_i: Compute r^{format}, r^{acc}, r^{bal} Aggregate advantage: A^b_i = (r^b_i - μ_r)/σ_r Policy update via clipped objective and KL penalty Distill outputs, apply Copy/Route mechanisms during SFT and inference

(Chu et al., 29 May 2025)

S f-AC Algorithm for Symmetric BRPO

Sample transitions (s, a, r, s') Update Q and value critics Advantage regression: w(s,a) Update ζ to approximate π_Reg via KL loss Sample b ~ π_θ(·|s), compute clipped ratio r(s,b) Compute conditional-symmetry loss by Taylor expansion Update θ via Adam: minimize KL loss + symmetry loss

(Zhu et al., 6 Aug 2025)

4. Baseline Strategies and Variance Reduction

BRPO introduces specialized baselines to reduce policy-gradient estimator variance:

• Budget-Relative Baseline $V_1(s_t)$: Aggregates past truncated summary rewards, discounted by $\lambda$, scaled by remaining budget weights, enabling better credit assignment for token-level actions as budgets increase.
• Group-Average Baseline $V_2(x)$: Empirical return normalization across independent chains, as in GRPO.
• Linear Interpolation: $V(s_t) = \alpha_t V_1 + (1-\alpha_t) V_2$, with $\alpha_t$ scheduling transition from group-based to budget-relative estimation.

Empirical analysis demonstrates elevated correlation between the interpolated baseline and true return, yielding reduced variance and more stable policy improvement. In vision-language settings, intra-group normalization of composite rewards balances accuracy, formatting, and reflective sequence optimality (Qi et al., 19 May 2025, Chu et al., 29 May 2025).

5. Integration with Decoupled and Composite Optimization

BRPO frameworks decouple the training of thinking (reasoning) and summary (output generation) policies:

• The summary policy $\pi_\phi$ is trained independently under its own budget prior $p'_\mathcal{B}$, ensuring robust summarization across truncation regimes.
• In reflective models, the policy implicitly encodes both "when" and "how" to reflect, guided by reward components penalizing excessive or insufficient reflection.
• In offline RL, the actor and critic networks are updated at separate timescales, with Taylor-series approximated regularization loss integrated for stable learning.

This decoupling supports domain robustness and permits controlled deployment in multi-objective environments.

6. Empirical Performance and Ablation Studies

BRPO consistently outperforms baselines such as GRPO, AWAC, and forward-KL regularization in token-efficient reasoning, visual QA, and offline RL control:

• On AMC22/AIME24, AR-uniform variant achieves 59% accuracy at 2000 tokens versus 53% for GRPO; all AR variants outperform GRPO even under skewed training budgets.
• In Qwen-LookAgain, BRPO with Visual Token COPY/ROUTE reduces hallucination (CHAIR_i down from 9.4% to 3.7%) while improving state-of-the-art VQA performance.
• In offline RL, S f-AC achieves competitive or superior results in MuJoCo benchmarks, with policies remaining well-centered, stable, and safe under symmetric regularization (Qi et al., 19 May 2025, Chu et al., 29 May 2025, Zhu et al., 6 Aug 2025).

Ablations reveal optimal settings for reflection frequency, baseline interpolation, and Taylor truncation orders yield strong trade-offs in accuracy, policy stability, and computational efficiency.

7. Limitations and Potential Directions

• BRPO inference cost scales with the number of budgets or reflection segments; scaling to fine granularity mandates efficient engineering solutions.
• Monotonicity of summary reward under token extension is assumed; in tasks with non-monotonic returns, baseline effectiveness may degrade.
• Domain-specific mechanisms (e.g., tree-structured attention, visual token re-injection) may constrain generalizability across architectures.
• Symmetric divergence approximation is robust for moderate truncation order; extremely high-order expansions can introduce instability in some settings.

Extensions suggest meta-learning budget priors, continuous or data-dependent budgeting, streaming inference, and generalized reflection balance can be incorporated to further advance BRPO's versatility and impact in reasoning under resource constraints.