Contextual Multi-Armed Bandits

• Contextual multi-armed bandits are sequential decision-making frameworks that adapt action selection using side-information to minimize context-dependent risks.
• They employ logistic models and uncertainty-guided exploration to efficiently learn low-regret policies in applications like autonomous driving and cyber-physical systems.
• The approach integrates statistical guarantees with practical monitoring to balance performance and safety through adaptive controller selection.

A contextual multi-armed bandit (CMAB) is a sequential decision-making framework that extends the classic multi-armed bandit by leveraging side-information (context) to adapt optimal action selection in each round. At each time step, the learner receives a context vector, chooses an arm (action) among a finite set, and observes a possibly stochastic reward or safety outcome. The goal is to learn a policy that maximizes reward (or minimizes cost/risk) conditioned on context. CMABs underpin modern context-driven monitoring and controller selection for autonomous and cyber-physical systems, particularly when operational safety or performance is context-dependent.

1. Formal Definition and Problem Structure

Let $A = \{c_1,\dots,c_n\}$ denote a finite set of arms (actions, controllers). On each round $t$:

• The learner observes a context $x_t \in \mathcal{X} \subseteq \mathbb{R}^d$.
• An arm $c_t \in A$ is selected according to a (possibly randomized) policy $\pi: \mathcal{X} \rightarrow A$.
• A stochastic outcome $Y_t \in \{0,1\}$ is realized, interpreted (in safety tasks) as a specification violation ($1$) or safe run ($0$); a reward formulation is $r(c,x) = 1-Y$.

The objective is to learn a policy $\pi$ minimizing the context-conditioned violation probability: $L(c,x) = \Pr[\text{violation} \mid c, x]$ so that

$\pi^*(x) \in \arg\min_{c \in A} L(c, x)$

Performance is measured by contextual regret: $$R_T = \sup_{x \in \mathcal{X}} \left[ L(\pi_T(x), x) - L(\pi^*(x), x)\right]$$ where $\pi_T$ is the policy after $T$ rounds (Luque-Cerpa et al., 28 Jan 2026).

2. Learning Algorithms and Statistical Models

A prevalent approach models $L(c,x)$ via arm-specific parameterizations. In logistic contextual bandits: $$\Pr[Y=1|c,x] = \sigma(\theta_c^\top x), \quad \sigma(z) = \frac{1}{1+e^{-z}}$$ Each arm $c$ has a parameter vector $\theta_c$. Estimation employs maximum likelihood, updating

$$\theta_{t,c} = \arg\max_{\theta \in \mathbb{R}^d} \sum_{s: c_s = c} \big[Y_s \log \sigma(\theta^\top x_s) + (1-Y_s)\log(1-\sigma(\theta^\top x_s))\big]$$

A principled exploration strategy is active learning via epistemic uncertainty—selecting the $(c,x)$ pair maximizing the context’s variance with respect to the Hessian $H_{t-1}^{(c)}$: $$(c_t, x_t) = \arg\max_{c \in A, x \in \mathcal{X}} \|x\|_{H_{t-1}^{(c), -1}}$$ Efficient updates use the Sherman–Morrison formula to maintain the inverse Hessian online (Luque-Cerpa et al., 28 Jan 2026).

This learning rule, coupled with MLE and uncertainty-guided sampling, yields high data efficiency and rapidly converges to low-regret policies, as established by the following regret bound:

Theoretical Result	Statement
Regret Bound	$R_T = O(\sqrt{\ln^2 T / T})$ (with high probability)
Parameter confidence	$$\|\theta_{t,c} - \theta^*_c\|_{H_{t,c}} \leq \beta_t, \quad \beta_t = O(\sqrt{d\ln(t/\delta)})$$

3. Context Representation and Feature Engineering

Contexts are encoded as fixed-dimensional vectors incorporating operational, environmental, or system-derived features relevant to safety or performance. For example, in simulated autonomous driving settings, context vectors may be constructed as:

• One-hot encoding of weather/time presets (ambient light, precipitation): 14 dims
• Binary indicators (e.g., intersection/road type): 1 dim
• Discretized proximity metrics (distance to nearest vehicle, pedestrian): 5–10 dims

These are concatenated and normalized in accordance with statistical assumptions underpinning the regret and confidence analyses. The resulting context space is typically bounded to ensure technical conditions for theoretical guarantees (Luque-Cerpa et al., 28 Jan 2026).

4. Applications in Context-Aware Runtime Monitoring

Contextual multi-armed bandits form the core of monitor-guided ensemble controller selection in AI-based autonomy. Rather than averaging controller predictions or votes—an approach that can dilute individual strengths and increase conservatism—CMAB-based runtime monitors route control authority based on the context, allocating to the controller empirically best suited to the present operational regime. This is essential in domains where controller risk varies sharply with context, such as:

• Autonomous driving under varying weather, road geometry, and traffic conditions
• Safety-critical cyber-physical systems with regime-dependent hazards

Experimental validation in simulation (e.g., Carla-based scenarios) demonstrates that contextual monitors built via active CMAB learning outperform averaging, mixture-of-experts, and black-box neural monitor approaches, especially in minimizing unnecessary fail-safe switches (false positives) and optimizing average safety reward:

Method	Avg Reward (Sc 1)	Avg Reward (Sc 2)
Ensemble	0.433	0.128
MoE	0.524	0.502
LR-Monitor (FP=30%)	0.817	0.717
NN-Monitor (FP=30%)	0.559	0.574

Moreover, increasing the number of controllers in the ensemble (from 1 to 15) yields monotonic gains in reward and reductions in fail-safe activation rates, validating the value of controller specialization exploited by the contextual monitoring paradigm (Luque-Cerpa et al., 28 Jan 2026).

5. Theoretical Guarantees, Limitations, and Data Efficiency

The main theoretical result is a high-probability uniform regret bound. Provided contexts and parameters are bounded, and outcomes are conditionally independent, contextual bandit monitors can guarantee that—in every context—the selected controller’s violation probability rapidly approaches that of the optimal controller, with convergence rate $O(\sqrt{\ln^2 T / T})$.

Nonetheless, limitations persist:

• The logistic risk model assumes linearity in feature-context mapping, which may be insufficient for highly nonlinear domains.
• Contexts are typically memoryless (no history), so stateful dependencies are missed.
• Discretization and feature engineering are required to encode complex operational regimes.

A plausible implication is that extending to deep (nonlinear) contextual bandits or predictive-state models may further enhance performance, particularly in high-dimensional or temporally correlated environments (Luque-Cerpa et al., 28 Jan 2026).

6. Integration with Formal Safety and Monitoring Architectures

CMABs offer a statistical foundation complementary to formal verification and temporal logic monitoring. For instance:

• In model-driven safety frameworks, context-aware monitors generated by STPA, STL, or Event Calculus can be triggered or configured adaptively via the current context estimate, which in turn may be learned via CMAB techniques.
• In runtime safety enforcement, a CMAB-style monitor can arbitrate between fast (possibly unverified) learned controllers and conservative fallback logic, maximizing performance while ensuring statistical safety guarantees.

This integration is especially relevant as AI-based controllers proliferate in safety-critical systems, and as monitoring requirements shift from static, rule-based guards to data-driven, context-sensitive deployment (Luque-Cerpa et al., 28 Jan 2026).

7. Outlook and Research Directions

Current research on contextual multi-armed bandits for monitoring emphasizes:

• Nonlinear and stateful extensions: deep CMABs, history-aware features
• Predictive context construction for proactive interventions
• Minimal data regimes: active learning to accelerate convergence under limited supervision

Open challenges include automating context feature extraction for highly complex environments and integrating CMAB-based runtime monitoring with verification stacks and certified fallback behavior. Anticipated developments are the use of CMABs as a universal substrate for context-adapted safety-critical control in complex, uncertain environments (Luque-Cerpa et al., 28 Jan 2026).