Online Model Selection in RL

• Online model selection in reinforcement learning is the process of adaptively choosing from a portfolio of models or hyperparameter settings to maximize cumulative rewards under finite interaction budgets.
• It employs meta-algorithms such as ESBAS, D³RB/ED²RB, and UCB variants to balance exploration and exploitation, particularly in nonstationary environments.
• Theoretical guarantees are provided via short-sighted and absolute pseudo-regret bounds, which underpin applications in hyperparameter tuning, transfer learning, and robust policy selection.

Online model selection in reinforcement learning (RL) refers to the adaptive and sequential process of selecting, among a portfolio of RL algorithms, models, or hyperparameter configurations, the one to deploy or allocate compute to at each decision point, in order to optimize cumulative performance under a finite interaction budget. This problem arises in both online and offline RL deployments, hyperparameter optimization via bandit-driven tuning, transfer learning via source-policy selection, and in settings with unknown or nonstationary environment dynamics. The core challenge is to efficiently allocate data and computation to candidate models so as to guarantee sample-efficient convergence to (or near) the best available policy, measured in cumulative reward, while controlling worst-case and instance-dependent regret.

1. Formal Problem Statement: Online Model Selection Paradigms

The canonical online model selection problem in RL is formalized by considering an episodic RL framework, with:

• A discrete or continuous state space $\mathcal{S}$, finite action set $\mathcal{A}$, reward function $R$, transition kernel $P$, initial distribution $\rho$, and episode horizon $H$.
• A finite portfolio $\mathcal{P} = \{\alpha^1, ..., \alpha^K\}$ of off-policy RL algorithms or model configurations, each capable of generating policies $\pi^\alpha$ using accumulated data $\mathcal{D}$.
• At each episode $\tau$, a meta-algorithm $\sigma$ chooses a model $\alpha = \sigma(\mathcal{D}_{\tau-1})$, whose policy $\pi^\alpha$ governs the agent, generating a trajectory $\varepsilon_\tau$ with (possibly discounted) return $\mu(\varepsilon_\tau)$.
• After $T$ episodes, the meta-algorithm is evaluated via its cumulative expected return:

$$\mathbb{E}_\sigma \left[\sum_{\tau=1}^T \mu(\varepsilon_\tau)\right] = \mathbb{E}_\sigma\left[\sum_{\tau=1}^T \mathbb{E}[\mu | \pi^{\sigma(\tau)}_{\mathcal{D}_{\tau-1}}]\right]$$

• Two primary metrics are defined:
  • Absolute pseudo-regret: measures loss to the optimal asymptotic model in the portfolio,

  $$\rho_{\mathrm{abs}}^\sigma(T) = T\mu^*_\infty - \mathbb{E}_\sigma\left[\sum_{\tau=1}^T \mathbb{E}[\mu(\pi^{\sigma(\tau)}_{\mathcal{D}_{\tau-1}})]\right]$$

  where $$\mu^*_\infty = \max_{\alpha \in \mathcal{P}} \lim_{|\mathcal{D}| \to \infty}\mathbb{E}[\mu(\pi^\alpha_{\mathcal{D}})]$$. - Short-sighted pseudo-regret: regret against the best algorithm at each episode,

  $$\rho_{\mathrm{ss}}^\sigma(T)=\mathbb{E}_\sigma \left[ \sum_{\tau=1}^T \left(\max_{\alpha\in\mathcal{P}} \mathbb{E}[\mu(\pi^\alpha_{\mathcal{D}_{\tau-1}})] - \mathbb{E}[\mu(\pi^{\sigma(\tau)}_{\mathcal{D}_{\tau-1}})] \right)\right]$$

This framework generalizes to online selection among pre-trained offline RL models, source policies in transfer, or base agents differentiated by architectures, hyperparameters, or random seeds (Laroche et al., 2017, Afshar et al., 1 Dec 2025, Li et al., 2023, Afshar et al., 2024, Li et al., 2017).

2. Meta-Algorithmic Approaches: Bandit-Based and Beyond

The prototypical methodology for online model selection is to formulate the scheduling of base algorithms as a nonstationary multi-armed bandit problem, where each arm corresponds to a candidate agent or configuration. The primary challenge is non-stationarity: each base learner's average return changes as it accumulates data and improves its policy.

Epochal Stochastic Bandit Algorithm Selection (ESBAS) (Laroche et al., 2017)

• ESBAS freezes the candidate policies at the start of exponentially growing epochs, using a fresh stochastic bandit (e.g. UCB1) per epoch to select among the policy snapshots.
• Within each epoch, since policies are fixed, model selection reduces to a classical stationary bandit problem.
• ESBAS achieves short-sighted pseudo-regret $O(\log^2 T/\Delta^\dagger_\infty)$ if reward gaps are bounded below, and absolute pseudo-regret that tracks the best base algorithm up to a constant factor and the bandit regret.

Sliding-Window and True Online Adaptation (SSBAS) (Laroche et al., 2017)

• SSBAS adapts ESBAS to a fully online setting by running a bandit with rewards computed over a sliding window for each base agent, allowing non-stationary adaption.

Data-Driven Regret-Balancing (D³RB/ED²RB) (Afshar et al., 1 Dec 2025, Pacchiano et al., 2023, Afshar et al., 2024)

• D³RB and ED²RB maintain, for each base agent $i$, estimates of a realized regret-coefficient $d^i_t$, tracking the scale of regret actually incurred.
• At each round, a potential $\phi^i_t = \hat d^i_t \sqrt{n^i_t}$ is used, and selection is made by pulling the least pulled (in this sense) agent.
• Potentials are doubled if the confidence interval for agent $i$ is breached, enabling robust adaptation under nonstationarity and misspecification.

UCB and Variants in Model/Policy Selection (Li et al., 2023, Li et al., 2017, Merentitis et al., 2019)

• Online UCB1 (or similar) is applied to the choice among a static set of models, policies, or agents, yielding $O(\sqrt{K T})$ cumulative regret and rapid convergence to the best available option, provided there exists an optimal arm.

Regret-Based Elimination and Model Selection for Function Approximation (Lee et al., 2020, Foster et al., 2019, Masoumian et al., 2024)

• Meta-algorithms use data-driven tests to eliminate models or function classes that incur statistically significant excess regret relative to more complex candidates.
• Regret bounds scale with the complexity of the simplest well-specified model and polynomially in the number of candidate classes, e.g., $\tilde O(M C^2_{m^*} B_{m^*}(T))$ in average-reward RL (Masoumian et al., 2024).

Table 1: Selected Meta-Algorithmic Approaches

Approach	Base Assumptions	Regret Guarantee
ESBAS/SSBAS	Off-policy, shared data	$O(\log^2T/\Delta)$ (short-sighted), tracks best algorithm (Laroche et al., 2017)
D³RB/ED²RB	Finite base, realized regret	$O(d_* M\sqrt{T})$ on realized regret (Afshar et al., 1 Dec 2025, Pacchiano et al., 2023)
UCB1 (model/policy)	Finite arms, stationary	$O(\sqrt{K T})$ (or $O(\log T)$ in favorable cases) (Li et al., 2017, Merentitis et al., 2019)
Regret elimination	Nested classes, model misspec.	$O(MC_{m^*}^2B_{m^*}(T))$ (Masoumian et al., 2024, Lee et al., 2020)

3. Theoretical Guarantees and Regret Analysis

Robust guarantees for online model selection are typically established under explicit assumptions about realizability (existence of a well-specified model/class in the portfolio), bounded reward gaps, or concentration of empirical rewards. Key results include:

• ESBAS: Short-sighted pseudo-regret $O(\log^2 T/\Delta^\dagger_\infty)$ under bounded gap, absolute pseudo-regret tracks any base up to a constant; anytime operation through epoch schedule (Laroche et al., 2017).
• D³RB / ED²RB: For $M$ base learners with realized regret coefficients $d^i_t$, guarantee

  $$\mathrm{Regret}(T) = O(d_* M\sqrt{T} + d_*^2\sqrt{M T})$$

  with $d_* = \min_{i} \max_{t\le T} d^i_t$ (Pacchiano et al., 2023, Afshar et al., 1 Dec 2025, Afshar et al., 2024).

• Regret elimination (ECE, MRBEAR): For $M$ candidate classes, ensures regret only $O(M)$ larger than the best base’s guarantee; for average-reward, $\tilde{O}(M C_{m^*}^2 B_{m^*}(T))$ (Masoumian et al., 2024, Lee et al., 2020).
• UCB/EXP3 variants: Standard UCB1 achieves $O(\log T)$ identification regret for stationary arms, $O(\sqrt{KT})$ in the worst-case; sliding-window/discounted variants accommodate nonstationarity at the cost of increased variance (Li et al., 2023, Merentitis et al., 2019).
• Adaptive allocation: Resource allocation fraction for agent $i$ adapts as

  $$\alpha^i_t = \frac{(1/d^i_t)^2}{\sum_{j}(1/d^j_t)^2}$$

  ensuring more sampling to better-performing bases (Afshar et al., 1 Dec 2025).

4. Handling Nonstationarity and Adaptation

Many RL settings manifest time-varying optimal model choices due to nonstationary task dynamics, policy improvement over time, or stochastic optimization variance. Several mechanisms are employed:

• Sliding-window bandit statistics: e.g. in SSBAS, restrict reward window to the most recent $W$ samples to track evolving arm means (Laroche et al., 2017).
• Dynamic or data-driven regret estimation: D³RB and ED²RB continuously update each base’s performance coefficient and doubling schedule to reflect changing regimes (Afshar et al., 1 Dec 2025, Pacchiano et al., 2023, Afshar et al., 2024).
• State-dependent dynamic model selection: Frame the entire model selection process as an RL problem, where the meta-policy $\pi_\mathrm{meta}$ switches among base models in response to covariate drift or incurred switching cost (Bellman-optimality in high-dimensional state/action MDPs) (Cordoni et al., 2023).
• Self-model selection: When some base agents are unreliable (e.g., different random seeds for deep RL), meta-selection can concentrate on the runs that achieve favorable trajectory returns, yielding higher-confidence learning (Afshar et al., 1 Dec 2025, Pacchiano et al., 2023).

5. Applications: Hyperparameter Tuning, Transfer, and Structured Model Selection

The techniques discussed are instantiated in a variety of RL application domains:

• Learning rate and hyperparameter adaptation: Treating different step sizes or optimizer settings as base agents, meta-algorithms like D³RB/ED²RB adaptively track the value of hyperparameters as optimization landscapes shift, outperforming fixed schedules and standard bandits in nonstationary regimes (Afshar et al., 2024).
• Neural architecture selection: Portfolio-based approaches allow dynamic allocation among deep network architectures, with selection schedules concentrating resources on high-capacity models as evidence warrants (Afshar et al., 1 Dec 2025).
• Source-policy selection in transfer: Model selection over a library of prior policies as discrete arms augments Q-learning with selective reuse, yielding theoretical and empirical gains in transfer learning (Li et al., 2017).
• Surrogate-augmented selection with sparse rewards: Auxiliary information gains or exploration bonuses act as surrogate rewards, smoothing early non-informative interaction and accelerating model identification (Merentitis et al., 2019).
• Average-reward RL and sequential games: MRBEAR demonstrates model selection in the average-reward (steady-state) regime with applications to repeated games with unknown partner memory and robust regret bounds (Masoumian et al., 2024).
• Model selection with functional approximation: Adaptive elimination meta-algorithms yield minimal loss over the optimal function class, with regret scaling near that of the oracle model (Lee et al., 2020, Foster et al., 2019).

6. Open Problems, Extensions, and Practical Considerations

Despite significant advances, several open challenges remain for online model selection in RL:

• Scalability to large model sets: Regret bounds are $O(M)$ in the number of candidates; reducing to $O(\log M)$ is partially addressed for linear bandits (ALEXP) (Kassraie et al., 2023) but open in general RL.
• Data sharing across base agents: Current frameworks often segregate replay data; leveraging cross-agent data via importance weighting or shared experience buffers could improve efficiency (Afshar et al., 1 Dec 2025).
• Continuous and structured model spaces: Extending regret-balancing or elimination strategies to continuous hyperparameter/model spaces (e.g., via Bayesian optimization as a base bandit) is an open direction (Merentitis et al., 2019).
• Instance-dependent and anytime guarantees: Refined analysis provides instance-dependent regret scaling with minimal overhead, and some approaches yield anytime/no-horizon guarantees (Kassraie et al., 2023).
• Function approximation and bootstrapping: Regret estimation in deep RL is noisy; variance-reduced or bootstrap confidence sequences are required to maintain sample efficiency at scale (Pacchiano et al., 2023).
• Switching costs and structured dependencies: Dynamic-programming meta-RL formulations explicitly model the tradeoff between model-switching costs and cumulative reward (Cordoni et al., 2023).
• Assumptions on stationarity and realizability: Most theoretical guarantees require that an optimal policy exists in the finite portfolio; in adversarial or highly nonstationary settings, methods may need explicit mixing or resetting (Pacchiano et al., 2023).

Online model selection in RL thus constitutes a critical link between theory and efficient, robust RL deployment in practice, enabling principled automated tuning, exploitation of transfer, and the robust deployment of RL agents in adaptive and nonstationary environments.