Boundary-Aware Policy Optimization

• Boundary-Aware Policy Optimization is a reinforcement learning method that integrates calibrated 'I DON'T KNOW' actions to manage reasoning boundaries in agentic systems.
• It employs group-based rewards and adaptive modulators to balance exploration with reliable refusal, preserving gradient stability during training.
• Variants like GRPO and adaptive clipping techniques have demonstrated improved reliability and reduced overconfident outputs on multi-hop QA benchmarks.

Boundary-Aware Policy Optimization (BAPO) encompasses a family of reinforcement learning (RL) techniques designed to address the challenge of boundary recognition in policy learning—especially within LLM-driven agentic systems and in RL settings with off-policy or safety constraints. Recent works on arXiv provide two prominent BAPO variants: Boundary-Aware Policy Optimization for reliable agentic search, which instills calibrated 'I DON'T KNOW' (IDK) responses at reasoning boundaries (Liu et al., 16 Jan 2026); and Balanced Policy Optimization with Adaptive Clipping, which ensures gradient stability and entropy preservation in off-policy RL for LLM alignment (Xi et al., 21 Oct 2025). These BAPO paradigms are distinct from generic safety-oriented approaches such as Proactive Constrained Policy Optimization (PCPO) (Yang et al., 3 Aug 2025), though they all share an emphasis on principled boundary awareness through reward, exploration, or clipping mechanisms.

1. Reinforcement Learning Formulation for Agentic Boundary Awareness

BAPO for reliable agentic search articulates the problem as an episodic Markov Decision Process (MDP) $(S, A, P, R, \gamma)$:

• States ($S$): Entire history of model-generated reasoning and current retrieval context (top-$k$ documents).
• Actions ($A$): At each step, the agent can either "think" (token generation for reasoning), "search" (external query emission), or take the "IDK" action (terminate with 'I DON'T KNOW').
• Transitions ($P$): Deterministic under generation; stochastic under search due to retrieved snippet randomness.
• Reward ($R$): Assigned per trajectory or per step, combining answer correctness (F1), answer format, and explicit IDK bonuses.
• Discount ($\gamma$): Set to $1.0$ in finite-horizon reasoning episodes.

A trajectory for a question $x$ of length $T$ is represented as:

$$\tau = ((r_1,a_1,o_1), \ldots,(r_{T-1},a_{T-1},o_{T-1}), r_T,y)$$

where $r_t$ denotes the reasoning prefix, $o_t$ the retrieved results, $a_t$ the action, and $y$ the answer.

2. Group-Based Boundary-Aware Reward Design

BAPO's reward structure aims to induce accurate boundary recognition without enabling trivial refusal behaviors. For $G$ rollouts per question under the current policy:

• Standard Correctness Reward:

$$\mathcal{R}^{\mathrm{Correct}}_i = \begin{cases} \mathrm{F1}(\hat y_i, y_{\mathrm{true}}) & \text{if format correct} \ -1 & \text{otherwise} \end{cases}$$

• Boundary Awareness: If no rollout among $G$ achieves positive correctness ($\mathcal{R}^{\mathrm{Correct}}_j > 0$ for all $j$ is false), then an additional IDK reward is assigned:

$\mathcal{R}^{\mathrm{IDK}}_i = \begin{cases} 0.5 & \text{if all $\mathcal{R}^{\mathrm{Correct}}_j \le 0$and$\hat y_i=$IDK} \ 0 & \text{otherwise} \end{cases}$

Thus, total reward per rollout: $$\mathcal{R}_i = \mathcal{R}^{\mathrm{Correct}}_i + \mathcal{R}^{\mathrm{IDK}}_i$$.

The group-based mechanism ensures that IDK is only rewarded if all rollouts across the group fail to reach the correct answer, thereby cultivating reliable refusal strictly at the boundary rather than as a shortcut.

3. Adaptive Reward Modulation and Exploration Control

Unconditional IDK rewards often lead to mass refusal ("IDK hacking"). BAPO integrates a time-dependent modulator $m(t) \in [0,1]$ that governs when IDK reward is available:

• During early exploration ($t \le T_{\mathrm{exp}}$), $m(t)=0$—IDK rewards are suppressed, enforcing exploration and answer attempts.
• Upon entering a plateau phase ($t > T_{\mathrm{exp}}$), $m(t)=1$ for low-diversity questions (no diversity in group rollouts), otherwise $m(t)=0$.

This dual-stage, per-question modulator enforces sample-level and stage-level gating, supporting balanced exploration and reliable refusal only when warranted by the underlying evidence or reasoning capacity. The net per-rollout reward is:

$$\mathcal{R}^{\mathrm{total}}_i = m(t)\,\mathcal{R}^{\mathrm{IDK}}_i + (1)\,\mathcal{R}^{\mathrm{Correct}}_i$$

4. Optimization Objectives: GRPO and BAPO Clipping

BAPO training employs Group Relative Policy Optimization (GRPO), a PPO-style objective adapted for grouped rollouts:

$$J(\theta) = \mathbb{E}_{x, \{\tau_i\}} \left[\frac{1}{G}\sum_{i=1}^G \min \left(w_i(\theta)A_i,\, \text{clip}(w_i(\theta), 1-\epsilon, 1+\epsilon)A_i \right)\right]$$

where $$w_i(\theta) = \frac{\pi_\theta(\tau_i|x)}{\pi_{\theta_\text{old}}(\tau_i|x)}$$ and $A_i$ is a normalized, standardized advantage.

The Balanced Policy Optimization variant of BAPO, designed for off-policy RL, addresses systematic instability and entropy collapse induced by fixed symmetric clipping. It adaptively sets asymmetric clipping bounds $(c_\text{low}, c_\text{high})$ to guarantee that positive-advantage tokens (reinforcement signals) constitute at least a fixed ratio $\rho_0$ of the gradient, thereby actively preserving exploration and reward diversity (Xi et al., 21 Oct 2025). The algorithm dynamically adjusts $(c_\text{low}, c_\text{high})$ per mini-batch to enforce balance, preventing negative-advantage domination and gradient explosion.

5. Empirical Results and Ablation Analyses

Experiments on four multi-hop QA benchmarks (HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle) demonstrate BAPO's marked gains in reliability. The Qwen2.5-7B-Instruct backbone, averaged across benchmarks, yields the following results:

Method	acc	prec	ρ_IDK	reliability
Search-R1	43.1	43.1	0.0%	43.1
ReSearch	50.0	50.0	0.0%	50.0
GRPO (vanilla)	59.3	59.3	3.8%	59.1
BAPO	55.5	64.3	16.7%	64.1

Across scales (3B, 7B, 14B), BAPO improves reliability by +13.9%, +9.7%, +11.9% over vanilla GRPO respectively, with less than 3-point accuracy sacrifice. Ablation studies isolate the contribution of boundary-aware reward and modulators:

Variant	acc	prec	ρ_IDK	reliability
Full BAPO	44.8	52.8	16.8%	51.3
– no boundary-aware reward	30.6	62.4	53.1%	44.8
– no sample-level modulator	43.3	52.0	20.4%	50.1
– no stage-level modulator	37.8	56.0	35.2%	49.0

The data confirm that boundary reward without modulators leads to excessive IDK responses and collapsed accuracy; stage- and sample-level modulators are critical for preserving agentic search exploration and stable refusal behavior (Liu et al., 16 Jan 2026).

For Balanced Policy Optimization with Adaptive Clipping, stability and sample efficiency are experimentally validated on AIME 2024/2025. Quantitative gains surpass open-source and proprietary benchmarks (SkyWork-OR1-7B/32B, o3-mini, Gemini-2.5-Flash-Thinking), and ablation confirms the necessity of asymmetric clipping and target positive ratio tuning for entropy preservation and gradient stability (Xi et al., 21 Oct 2025).

6. Comparison to Boundary-Constrained and Safety-Oriented Policy Optimization

Proactive Constrained Policy Optimization with Preemptive Penalty (PCPO) targets safe RL with explicit constraints. Similar to BAPO, PCPO deploys a boundary-aware intrinsic reward, but its primary mechanism is a preemptive log-barrier penalty incorporated as the policy approaches constraint boundaries, guiding exploration away from feasibility limits. The methodology is mathematically distinct: PCPO solves a trust-region constrained MDP objective with log-barriers and an intrinsic reward term that activates as constraint violation nears (Yang et al., 3 Aug 2025). Theoretical analysis establishes duality-gap bounds and update performance, with empirical results confirming superiority over classic Lagrangian remedies in reducing oscillations, violation rates, and instability.

This suggests that while both BAPO and PCPO leverage boundary awareness, BAPO (in both reliable refusal and balanced off-policy RL) is primarily concerned with agentic self-awareness and entropy balance, rather than formal safety constraints. A plausible implication is that future RL frameworks may benefit from cross-pollination, integrating group-based refusal, adaptive modulators, and proactive barrier methods for generalized boundary-sensitive policy learning.

7. Practical Considerations and Theoretical Significance

BAPO introduces minimal additional hyper-parameters: $\rho_0$ (minimum positive loss share) and clipping ranges for Balanced Policy Optimization, stage/sample modulator thresholds and rollout group size $G$ for agentic search. The mechanisms are well-suited to PPO-style RL loops and are validated across backbone scales and rollout settings. Entropy preservation and gradient stability are theoretically certified via the Entropy-Clip Rule and covariance analysis for adaptive clipping (Xi et al., 21 Oct 2025).

In summary, BAPO methodologies:

• Enable reliable and calibrated 'I DON'T KNOW' responses in LLM agents strictly at reasoning boundaries, reducing the risk of plausible but unwarranted outputs.
• Stabilize off-policy RL optimization, prevent entropy collapse, and maintain sample efficiency in large-scale LLM alignment tasks.
• Serve as a principled RL recipe—novel reward composition plus adaptive gating or clipping—for injecting self-awareness and robustness into policy optimization.

These advances establish BAPO as a central technique for reliable, boundary-sensitive agentic RL, with implications for safe autonomous decision-making, robust exploration, and alignment in high-dimensional LLM domains (Liu et al., 16 Jan 2026, Xi et al., 21 Oct 2025).