Batched Contextual Bandit Learning

• Batched contextual bandit learning is a sequential decision-making framework where agents choose actions in batches and update their policies only after delayed feedback.
• The regret analysis shows that delayed updates trade off adaptivity for efficiency, with performance varying across linear, nonparametric, and neural models.
• Algorithmic paradigms like frozen-history, successive elimination, and dynamic batch sizing enable efficient exploration and exploitation in practical applications.

Batched contextual bandit learning refers to a broad class of sequential decision-making problems in which a decision maker receives contextual information and must select actions (arms) with limited or delayed feedback, structured in discrete batches. Unlike the classical contextual bandit setting—where feedback arrives immediately after each action—batched contextual bandits restrict updates to occur only at predefined intervals; decisions within batches are made without observing intermediate rewards. This framework is ubiquitous in domains where real-time feedback is unavailable or costly, including clinical trials, recommender systems, large-scale online experiments, and reinforcement learning from logged data. The batch structure introduces novel statistical and algorithmic phenomena, fundamentally trading off adaptivity for practical efficiency, and has generated a diverse literature ranging from linear and high-dimensional models to nonparametric, semi-parametric, and neural function classes.

1. Formalization and Canonical Problem Classes

The canonical batched contextual bandit problem considers a time horizon $T$ partitioned into $M$ batches with endpoints $0=t_0<t_1<\dots<t_M=T$. At each round $t$, the agent observes context $X_t$ (possibly in $\mathbb{R}^d$ or general $\mathcal{X}$), selects action $A_t$ from a finite or structured set, and receives a stochastic reward $Y_t=Y_t^{(A_t)}$ after the batch concludes. The goal is to minimize cumulative regret, typically

$$R_T = \mathbb{E}\left[\sum_{t=1}^T\left(\max_a f^a(X_t) - f^{A_t}(X_t)\right)\right],$$

where $f^a$ may be linear, nonlinear, or nonparametric in the context.

Feedback in this framework is constrained such that rewards for rounds $t_{m-1}< t\leq t_m$ are only revealed at $t_m$, precluding within-batch adaptivity. Actions within each batch are chosen based only on preceding badges' data and current contexts observed so far. Batched approaches interpolate between fully online learning ($M=T$) and pure offline policy evaluation ($M=1$).

Key problem classes include:

• Linear bandits and sparse/structured variants: Reward functions are linear or (group)-sparse in the context, possibly with high dimension ($d\gg 1$), e.g. (Fan et al., 2023, Ren et al., 2020, Swiers et al., 2024).
• Nonparametric contextual bandits: Rewards $f^a(x)$ are $L$-smooth in some (Hölder, Lipschitz) sense, supporting a rich covariate structure (Jiang et al., 2024, Arya et al., 1 Mar 2025).
• Semi-parametric models: Global structure is imposed via shared single-index models $f^a(x)=g^a(x^\top\beta_0)$, yielding sharp dimension-reduced regret rates (Arya et al., 1 Mar 2025).
• Neural/Kernelized bandits: Function class $f^a(x)$ is parameterized by a (possibly overparameterized) neural network, analyzed via neural tangent kernels (Gu et al., 2021).
• Parallel/Simultaneous action selection: Batched settings generalized to parallel environments or large-scale experimentation (multiple arms/patients per batch) (Chan et al., 2021).

2. Regret Analysis and Complexity of Batching

Batched feedback fundamentally degrades the achievable regret relative to online learning but, remarkably, the extent of this degradation is controlled by the batch count $M$ and function class assumptions.

• Linear bandits: With $K$ arms ($K<\infty$), regret typically increases by a multiplicative $\sqrt{b}$ or $b$ factor: for batch size $b=T/M$, regret is $R_T=O(\sqrt{bT})$ or $O(b\sqrt{T/b})$ under optimistic or Thompson-sampling policies, as established in (Provodin et al., 2022, Han et al., 2020).
• Adversarial/stochastic contexts: In adversarial settings, achieving online-optimal regret requires $M=\Omega(\sqrt{dT})$ batches; in stochastic (i.i.d.) contexts, only $M=O(\log\log T)$ batches suffice for fully adaptive performance (Han et al., 2020).
• Nonparametric/semiparametric settings: For $(\alpha,\beta)$-smooth reward functions in $d$ dimensions, the minimax regret scales as $T^{1-\gamma}$ in the fully adaptive regime (with $\gamma = \beta(1+\alpha)/(2\beta+d)$), but with only $M$ batches, the optimal exponent degrades to $(1-\gamma)/(1-\gamma^M)$, with $O(\log\log T)$ batches restoring adaptivity (Jiang et al., 2024, Arya et al., 1 Mar 2025).
• High-dimensional/sparse settings: Regret rates $\tilde O(s_0^2\log^2T)$ (with $s_0$ sparsity) are achievable with $O(\log T)$ batches for sparse linear bandits, and similarly for low-rank matrix models (Fan et al., 2023).
• Kernelized/neural bandits: For general nonlinear reward models parameterized via, e.g., neural tangent kernels, regret scales as $\tilde O(\tilde{d}\sqrt{T} + \tilde{d}T/B)$, where $\tilde{d}$ is an effective dimension tied to the kernel and data (Gu et al., 2021).

Theoretical lower bounds confirm these rates are sharp: batching induces unavoidable regret inflation unless $M$ is sufficiently large.

3. Algorithmic Paradigms

Batched contextual bandit algorithms adapt canonical online learning techniques for delayed feedback and restricted adaptivity:

• Frozen-history principle: Within a batch, the policy is held fixed, using parameters estimated at the previous batch boundary. This is observed in batched variants of LinUCB, LinTS, greedy LASSO policies, and neural UCB (Provodin et al., 2022, Fan et al., 2023, Gu et al., 2021).
• Successive elimination and binning: For nonparametric and semi-parametric conditional mean estimation, dynamic partitioning of the covariate space, with successive arm elimination within bins, enables minimax-adaptive rates. Key is choosing bin widths and split factors to balance exploration and exploitation per batch (Jiang et al., 2024, Arya et al., 1 Mar 2025).
• Two-stage sample allocation: Forced exploration in early batches, possibly via randomized assignment or uniform sampling, is deployed to guarantee estimator consistency, followed by exploitation steps with refined estimators (Fan et al., 2023, Ren et al., 2020).
• Dynamic/adaptive batch sizing: Several works optimize batch boundaries dynamically as a function of estimation error, cumulative regret, or batch costs, rather than fixing batch (or phase) sizes in advance (Fan et al., 2023, Ren et al., 2020).
• Parallelization schemes: Algorithms such as Parallel LinUCB and Parallel LinTS select a batch of actions using confidence sets or Thompson-sampling, possibly inserting deterministic or random diversity to minimize regret burn-in (Chan et al., 2021).
• Feature selection for sparsity and fairness: In high-dimensional settings, sequential inclusion of features with uncertainty-based thresholds (e.g., via z-scores on posterior mean estimates) controls both regret and fairness by excluding features until their impact is confidently established (Swiers et al., 2024).
• Quadratic programming for inverse problems: Estimating unknown reward and policy parameters from behavioral evolution histories in batched imitation learning can be formulated as tractable quadratic programs that incorporate both deterministic and randomized policies (Xu et al., 2024).

4. Extensions and Inference in Batched Bandits

Batched contextual bandits motivate methodological developments in statistical inference, design, and control:

• Finite-sample inference for adaptively collected batched data: The ordinary least squares estimator is not asymptotically normal under bandit data collection; the Batched OLS (BOLS) estimator achieves robust asymptotic normality with explicit per-batch weighting, even under nonstationary reward baselines and adaptive assignment (Zhang et al., 2020).
• Fairness and interpretability: Algorithms that control for irrelevant features—e.g., through sequential inclusion—yield fairer policies in the sense that irrelevancies are likely to be excluded from influencing decisions, and enable principled fairness regret metrics. This has direct implications for fairness-aware recommendation and decision-making systems (Swiers et al., 2024).
• Behavioral evolution and imitation learning: The Inverse Batched Contextual Bandit (IBCB) framework enables efficient estimation of environment reward parameters and expert policy as they evolve from novice to experienced status in streaming application settings, outperforming classical imitation learning in both empirical risk and generalization (Xu et al., 2024).
• Reinforcement fine-tuning as a batched contextual bandit: Recent work formalizes RLHF-style reinforcement fine-tuning in LLMs as a batched contextual bandit process, allowing precise experimental disentanglement of design choices such as rollout count, batch size, and advantage estimation (Xie et al., 30 Jan 2026).

5. Empirical Evidence and Applications

Empirical validation of batched contextual bandit algorithms spans diverse domains:

Application domain	Characteristic batch regime	Citation
Clinical trials	Small phase counts, adaptive cohort assignment	(Han et al., 2020, Jiang et al., 2024)
Recommender systems	Batched reward logs, real-time score updates	(Xu et al., 2024, Fan et al., 2023)
Online advertising	Parallel batch allocation, high-dimensionality	(Chan et al., 2021, Fan et al., 2023)
Crowdsourcing	Batches of tasks; binning for worker assignment	(Han et al., 2020, Gu et al., 2021)
LLM RLHF fine-tuning	Batched rollouts, stochastic rewards	(Xie et al., 30 Jan 2026)

BatchNeuralUCB demonstrates near-optimal regret in complex reward environments with drastic reduction in policy updates (Gu et al., 2021). In high-dimensional settings, batched greedy LASSO and sequential inclusion algorithms achieve oracle-level regret and fairness, with minimal computation cost relative to retraining-based baselines (Fan et al., 2023, Swiers et al., 2024). BIDS leverages a learned or hypothesized reward index to defeat the curse of dimensionality with orders-of-magnitude improvements in regret, as shown in both simulated and real-world datasets (Arya et al., 1 Mar 2025).

6. Batch Complexity, Phase Transitions, and Design Principles

A central result across the literature is the existence of phase transitions in batch complexity:

• Fully online adaptive performance is achievable with only $O(\log\log T)$ batches in stochastic i.i.d. or smooth nonparametric settings, and with $O(\log T)$ batches in high-dimensional sparse/low-rank problems, provided careful batch allocation and dynamic exploration (Han et al., 2020, Jiang et al., 2024, Fan et al., 2023).
• Adversarial setting forces at least polynomially many batches for near-optimality.
• Static greedy policies (e.g., no forced exploration) can be minimax-optimal in the high-dimensional sparse linear case with adaptive batch sizes (Ren et al., 2020).

Algorithmic guidelines include:

• Free adaptivity (small $M$) is attainable with judicious dynamic binning and progression of partition granularity per batch (Jiang et al., 2024).
• In high-dimensional regimes, greedy and forced-sampling strategies are sufficient to guarantee sparse model recovery and low regret, with strong empirical evidence across scientific and commercial applications (Fan et al., 2023, Ren et al., 2020).
• Batch size selection must consider both statistical regret and engineering cost, with practical batched deployments benefitting from batch sizes scaling as $T^{1/3}$ or $O(\log\log T)$, depending on domain constraints (Provodin et al., 2022).

7. Open Problems and Future Directions

Major directions for ongoing investigation include:

• Adaptive algorithms that optimally allocate batches without knowledge of smoothness or margin parameters.
• Nonparametric, sparse, or nonlinear function classes under adversarial context generation.
• Joint estimation and control of reward index directions in unknown, high-dimensional single-index or multi-index models.
• Theoretical understanding of batch-induced performance ceilings in deep RLHF and foundation model fine-tuning (Xie et al., 30 Jan 2026).
• Robustness of batched algorithms to non-Gaussian, heavy-tailed, or heteroskedastic reward noise.
• Minimizing statistical cost of batching in real-time, safety-critical applications (e.g., active drug trials).
• Designing statistically valid and efficient inference procedures for adaptively sampled batched bandit data (Zhang et al., 2020).

Batched contextual bandit learning thus occupies a central position in modern sequential learning theory, algorithmics, and applications—serving as a bridge between the theoretical foundations of adaptivity and the practical imperatives of efficient, scalable decision-making under real-world constraints.