On Upper-Confidence Bound Policies for Non-Stationary Bandit Problems

Abstract: Multi-armed bandit problems are considered as a paradigm of the trade-off between exploring the environment to find profitable actions and exploiting what is already known. In the stationary case, the distributions of the rewards do not change in time, Upper-Confidence Bound (UCB) policies have been shown to be rate optimal. A challenging variant of the MABP is the non-stationary bandit problem where the gambler must decide which arm to play while facing the possibility of a changing environment. In this paper, we consider the situation where the distributions of rewards remain constant over epochs and change at unknown time instants. We analyze two algorithms: the discounted UCB and the sliding-window UCB. We establish for these two algorithms an upper-bound for the expected regret by upper-bounding the expectation of the number of times a suboptimal arm is played. For that purpose, we derive a Hoeffding type inequality for self normalized deviations with a random number of summands. We establish a lower-bound for the regret in presence of abrupt changes in the arms reward distributions. We show that the discounted UCB and the sliding-window UCB both match the lower-bound up to a logarithmic factor.

• The paper introduces discounted and sliding-window UCB algorithms that adapt to changing reward distributions in non-stationary environments.
• It derives tight theoretical upper and lower regret bounds, demonstrating near-optimal performance under abrupt changes.
• Empirical evaluations show that both algorithms outperform standard UCB and EXP3.S in dynamic settings.

Analysis of "On Upper-Confidence Bound Policies for Non-Stationary Bandit Problems"

Introduction

The paper under discussion addresses the challenge of non-stationary multi-armed bandit problems (NS-MABPs), where the reward distributions associated with each arm can change over time. The focus is on adapting Upper-Confidence Bound (UCB) policies for these non-stationary environments. It introduces two algorithms, the discounted UCB and the sliding-window UCB, and analyzes their performance in terms of regret, establishing that they closely match the theoretical lower bounds for regret in non-stationary settings.

Methodology

The primary contributions are the adaptations of UCB algorithms to handle non-stationarity:

1. Discounted UCB: This approach uses a discount factor $\gamma$ to give more weight to recent rewards when estimating the mean reward of each arm. The UCB is constructed from these discounted averages, leading to a dynamic adaptation as the environment changes.
2. Sliding-Window UCB: Instead of discounting, this algorithm maintains and averages only the most recent $\tau$ observations for mean estimation. This localized view inherently responds to changes by forgetting older data.

Theoretical Results

The paper provides rigorous theoretical analyses for these algorithms:

• Upper Bound for Regret: For both algorithms, the paper derives upper bounds on the expected regret, showing they perform nearly optimally (up to logarithmic factors) under abrupt changes in reward distributions.
• Lower Bound on Regret: A lower bound is established for any algorithm in abruptly changing environments, illustrating the inherent difficulty of the NS-MABP and the near-optimality of the proposed methods.
• Deviation Inequality: A novel deviation inequality for self-normalized averages is introduced, addressing the complexities introduced by random summands in non-stationary settings.

Empirical Evaluation

Simulations demonstrate the practical effectiveness of the proposed algorithms:

• Adaptability: Both discounted and sliding-window UCB adjust more swiftly to changes compared to standard UCB and alternative policies like EXP3.S, minimizing regret in dynamic environments.
• Performance: The experiments confirm that the proposed methods outperform the baselines in scenarios with sudden and continuous changes in reward distributions.

Conclusion

The adaptations to UCB policies detailed in this paper provide robust solutions to NS-MABPs by leveraging recent rewards or localized data. This work highlights the critical balance between exploration and exploitation in dynamic environments, emphasizing the necessity of adaptive mechanisms. Future research could focus on optimizing the choice of the discount factor or window size in a data-driven manner to further enhance performance in varied non-stationary contexts.