On-Policyness Weighting for RL and Bandits

• The paper introduces adaptive, balance-based, and projected weighting methods that stabilize off-policy evaluation by prioritizing data with stronger alignment to the target policy.
• It leverages convex optimization and variance proxies to construct weights, thereby reducing the high variance associated with traditional inverse-propensity approaches.
• Empirical results demonstrate significant RMSE reduction and improved inferential coverage across contextual bandits and RL benchmarks, confirming robust performance gains.

On-policyness-based weighting encompasses modern methodologies for stabilizing and improving off-policy evaluation and learning from adaptively collected or offline data in contextual bandits and reinforcement learning. These approaches select or learn weights for each data point according to how closely its context and action match the distribution induced by a target policy—rather than relying on direct propensity ratios alone—thereby optimizing the so-called “on-policyness” of the weighted sample. This weighting paradigm reduces estimator variance, improves support coverage, and enables minimax mean-square-error optimization, as formalized in recent work (Zhan et al., 2021, Kallus, 2017, Wang et al., 2021).

1. Problem Formulation and the Need for On-Policyness-Based Weighting

Off-policy evaluation (OPE) refers to estimating the value of a target policy $\pi$ given data generated by a possibly different policy (logging or behavior policy). In contextual bandits, one observes data $(x_t, w_t, y_t)$ with context $x_t$, action $w_t \sim \pi_0(\cdot|x_t)$, and reward $y_t$. The analogous RL setting concerns discounted visitation distributions over state-action pairs. Naive inverse-propensity weighting (IPW) or doubly robust (DR) estimators normalize each sample via ratios such as $\frac{\pi(w_t|x_t)}{\pi_0(w_t|x_t)}$, which can introduce severe variance and instability if the target and logging policies have poor overlap. High-variance weights arise particularly when the logging probabilities $e_t(x,w)$ are small relative to the target, yielding “exploding” importance ratios in finite samples (Zhan et al., 2021, Kallus, 2017).

On-policyness-based weighting directly addresses this pathology by constructing weights that favor data points most consistent with the target policy, either through adaptive variance-stabilizing transformations (Zhan et al., 2021), direct balance optimization (Kallus, 2017), or state-action marginal balancing (Wang et al., 2021).

2. Methodologies for Weight Construction

Adaptive Weighting in Contextual Bandits

Zhan et al. (Zhan et al., 2021) develop an adaptive weighting scheme atop the DR estimator, using data-driven proxies for per-sample variance. For a target policy $\pi$, the weighting factor is given by $$h_t(x) = \phi\left(\sum_w \frac{\pi^2(w|x)}{e_t(x,w)}\right)$$, with $\phi$ chosen to be either inverse standard deviation ($1/\sqrt{v}$, "StableVar") or inverse variance ($1/v$, "MinVar"). The improved estimator

$$\widehat V^{AWDR}_T(\pi) = \sum_{t=1}^T \frac{h_t(x_t)}{\sum_{s=1}^T h_s(x_t)} \Gamma_t(x_t, \pi)$$

sharply downweights high-variance, low-overlap samples, directly reducing estimator MSE and yielding stable inference.

Balanced Weight Optimization

Kallus (Kallus, 2017) formulates weight selection as a convex optimization minimizing worst-case or posterior conditional mean squared error (CMSE), subject to sample balance between the target and historical distribution. For i.i.d. data $(X_i, T_i, Y_i)$ and policy $\pi$, the optimal weight vector solves

$$W^*(\pi) = \arg\min_{W\geq 0, \sum W_i = n}\; \sup_{\|f\|\leq 1} B^2(W,\pi;f) + \frac{1}{n^2} W^\top \Lambda W$$

where $B(W, \pi; f)$ measures discrepancies in function moments between weighted empirical data and target policy, and $\Lambda$ penalizes variance. These “balance-based weights” expand support beyond IPW and yield lower finite-sample variance.

Projected State-Action Balancing in RL

Wang et al. (Wang et al., 2021) advance a projected balancing approach in RL, where the optimal weights $w_{i,t}$ approximately satisfy moment-matching equations motivated by the Bellman backward recursion for discounted visitation. The optimization enforces, for each chosen basis function $B_k$,

$$\left| \frac{1}{nT} \sum_{i,t} w_{i,t} \hat{L}_k(S_{i,t},A_{i,t}) - \ell_k \right| \leq \delta_k$$

where $\hat{L}_k$ encodes local Bellman error and $\ell_k$ is the expected moment under $\pi$. This yields a quadratic program whose solution smoothly approximates the ideal marginal ratio $\omega^\pi(s,a)$, further regularized to prevent extreme weights.

3. Theoretical Guarantees and Finite-Sample Behavior

On-policyness-based weighting frameworks possess robust theoretical guarantees. In contextual bandits, adaptive weighting yields central limit theorems (CLT) for the studentized estimator under regularity conditions (bounded rewards, sufficient exploration, regression consistency, stability of adaptive weights). One obtains asymptotically normal t-statistics and valid confidence intervals of the form

$$\frac{\widehat V_T - V(\pi)}{\widehat{\sigma}_T} \xrightarrow{d} \mathcal{N}(0,1)$$

enabling inferential validity even under adaptive data collection (Zhan et al., 2021). Balanced weighting possesses minimax consistency and uniform regret bounds, with error rates scaling as $\mathfrak{R}_n(\Pi) + n^{-1/2}$ in policy learning (where $\mathfrak{R}$ is Rademacher complexity) (Kallus, 2017). In RL, projected balancing yields semiparametric efficient estimates, with convergence rates depending on basis representation and the minimal eigenvalue of the balancing Gram matrix (Wang et al., 2021). Coverage and absolute continuity conditions ($\sup_{s,a} \bar d^\pi(s,a)/\bar p^b_T(s,a)<\infty$) are necessary for well-posedness of the off-policy evaluation problem.

4. Comparative Analysis: On-Policyness vs. Classical Weighting

Classical IPW can discard large portions of data when target policy support does not align and can suffer explosive variance; balanced and adaptive weighting, by contrast, constructs weights with wider support and explicitly penalizes variance rather than resorting to ad hoc clipping (Kallus, 2017). In RL, marginal importance ratios can be less extreme than per-step product ratios; projected balancing shrinks outlier weights by moment-matching on rich function classes, preventing instability (Wang et al., 2021).

The following table contrasts salient features of three weighting paradigms:

Weighting Approach	Support Coverage	Variance Control
Inverse-Propensity (IPW/DR)	Sparse, often drops	None (ad hoc clipping)
Adaptive/Balance-based	Wide, all samples	Explicit via objective
Projected State-Action Balancing	Wide, smooth	Moment-matching penalty

5. Algorithmic Implementation

All major schemes can be cast as tractable convex programs:

• Adaptive weighting: $O(K)$ per step to compute variance proxies, with a single pass over the data (Zhan et al., 2021).
• Balance-based weights: Inner QP in $n$ variables, outer policy optimization via quasi-Newton iteration (Kallus, 2017).
• Projected balancing RL: Small-scale QP in $nT$ variables, with basis expansion and regularized regression steps (Wang et al., 2021).

Pseudocode is provided for each method, with key steps involving proxy/constraint computation, QP or convex dual solvers, and output value/inference statistics.

6. Empirical Performance and Applications

Across diverse domains (synthetic contextual bandit experiments, UCI datasets, RL benchmarks such as CartPole and mHealth), on-policyness-based weighting consistently yields lower RMSE, tighter standard errors, and improved inferential coverage compared to naive DR or IPW. For example, contextual MinVar AWDR cuts RMSE by 30–50% and yields near-nominal coverage (Zhan et al., 2021); balance-based learners outperform IPW and POEM methods in regret and support coverage by 5–15% (Kallus, 2017); projected balancing yields the lowest root-MSE and most plausible biological ordering in mHealth policy ranking (Wang et al., 2021). Contextual versions outperform aggregate approaches, and more aggressive variance penalization (MinVar) trades off slight bias for further error reduction.

7. Significance, Current Limitations, and Future Directions

The “on-policyness” paradigm leverages data-driven estimates of policy alignment to produce more statistically efficient, stable, and widely applicable policy evaluation and learning procedures. Such methods are central as datasets grow larger, policies more adaptive, and off-policy deployment more routine. Current challenges include basis selection for balancing, scaling optimization to extreme sample sizes, and optimal regularization for finite-sample trade-offs. Theoretical boundaries of estimator efficiency, coverage, and regret under worst-case overlap remain active areas of investigation. Approaches integrating balancing, adaptive variance control, and nonparametric estimation are likely to expand in importance as both contextual bandit and RL evaluation frameworks evolve.