Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation

Abstract: We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained $Q$-function, the learner aims to adapt it to the target environment using only a limited amount of online interaction. We first characterize the difficulty of this setting by establishing a minimax lower bound, showing that even when the pretrained $Q$-function is close to optimal $Q^\star$, online adaptation can be no more efficient than pure online RL on certain hard instances. On the positive side, under a novel structural condition on the offline-pretrained value functions, we propose O2O-LSVI, an adaptation algorithm with problem-dependent sample complexity that provably improves over pure online RL. Finally, we complement our theory with neural-network experiments that demonstrate the practical effectiveness of the proposed method.

• The paper establishes a minimax lower bound showing that without structural guarantees, an offline-trained Q-function does not reduce the online sample complexity compared to pure online RL.
• It introduces O2O-LSVI, a least-squares value iteration variant that leverages a β-separable value gap condition to selectively trust the offline Q-function, ensuring efficient adaptation.
• Empirical evaluations on MuJoCo AntMaze benchmarks demonstrate that the algorithm outperforms existing baselines and achieves significant reductions in required environmental interactions.

Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation

Problem Formulation and Theoretical Foundations

This work studies the efficiency limitations and potential of offline-to-online reinforcement learning (RL) when initialized with an offline-pretrained $Q$-function, within environments allowing general function approximation. The authors formally consider the setting in which a policy learner starts from a potentially imperfect $Q$-function, $Q_{ref}$, obtained via prior offline (batch) training, and seeks to adapt and improve it using a limited budget for online environment interactions.

A fundamental challenge in the offline-to-online paradigm is that the offline-learned values may lack sufficient coverage and generalization, particularly in regions with poor support or distribution shift. The paper rigorously quantifies the adaptation difficulty by establishing a minimax lower bound, demonstrating that—absent structural assumptions—even a reference $Q_{ref}$ arbitrarily close to optimal $Q^\star$ offers no significant reduction in online interaction complexity compared to pure online RL, for nontrivial MDP and function class instances.

Minimax Lower Bound: Adaptation Is Not Always Easier

The authors introduce a minimax lower bound construction using linear MDPs, proving that for any offline-to-online adaptation algorithm, the required number of online episodes to reach $\epsilon$-optimality is $\Omega(H^3 d^2 / \epsilon^2)$—matching the lower bound for pure online RL [(2604.13966), see Theorem~\ref{thm:lb}]. Notably, this holds even when $Q_{ref}$ is uniformly $\zeta H$-close to $Q^\star$ (with arbitrarily small $Q$0); i.e., value adaptation can be as hard as learning from scratch. This hardness result is not a consequence of model misspecification, as $Q$1 remains realizable within the function class, but rather of the inability to exploit $Q$2 in pathological cases lacking further structure.

This finding parallels earlier minimax analyses of policy or dynamic-model transfer in tabular and linear MDPs but extends to general function approximation settings and emphasizes the irrelevance of mere proximity between $Q$3 and $Q$4 in the absence of structural guarantees.

O2O-LSVI: Structural Conditions Enabling Sample-Efficient Adaptation

To overcome this limitation, the paper introduces a novel, model-free structural condition termed $Q$5-separable value gaps. This assumption asserts that each state-action-time triple satisfies either $Q$6 or $Q$7. The intuition is that $Q$8 is accurate on a large region but may be distinctly inaccurate on a minority region. The "accuracy coverage" parameter $Q$9 upper bounds the expected occupancy-weighted mass of the inaccurate regions under arbitrary policies.

The core algorithm, O2O-LSVI, is a variant of least-squares value iteration (LSVI) extended to general (possibly nonlinear) function approximation. It alternates between performing optimistic Bellman regression with UCB-style bonus for uncertainty-driven exploration, and selectively trusting $Q_{ref}$0 in regions where the learned confidence intervals certify its accuracy within the $Q_{ref}$1 band. This trust criterion is based on whether the constructed upper and lower bounds on $Q_{ref}$2 at a given $Q_{ref}$3 reside within a $Q_{ref}$4-neighborhood of $Q_{ref}$5. When this holds, O2O-LSVI directly sets its estimate to $Q_{ref}$6, otherwise exploring with optimism.

The theoretical analysis combines the generalized Eluder dimension (ED) as a complexity metric [Agarwal et al., 2023], bonus oracles for function class uncertainty, and UCB-based induction to yield both regret and sample complexity guarantees.

Regret and Sample Complexity Guarantees

Under Bellman completeness and bounded accuracy/misspecified coverage, O2O-LSVI exhibits regret: $Q_{ref}$7 where $Q_{ref}$8 is the Eluder dimension and $Q_{ref}$9 is the covering number of the function class [(2604.13966), Theorem~\ref{thm:upper}]. The sample complexity to obtain an $Q_{ref}$0-optimal policy can be as low as $Q_{ref}$1 in favorable regimes (i.e., small $Q_{ref}$2, large $Q_{ref}$3), compared to $Q_{ref}$4 for pure online RL.

When $Q_{ref}$5, adaptation offers a $Q_{ref}$6-factor improvement, translating into potentially orders-of-magnitude reduction in environmental interactions for well-structured transfer settings. The bounds achieve sharper dependence on $Q_{ref}$7 over prior tabular/domain-shift analyses (cf. [qu2024hybrid]).

When only a misspecified $Q_{ref}$8 is available (maximum error $Q_{ref}$9), an additional $Q^\star$0 regret term is provably unavoidable, highlighting the importance of high-quality pretraining or detection of misspecification.

Relation to Prior Work

This theoretical framework subsumes and extends both hybrid RL (combining offline and online data) and value iteration with Bellman-complete representation classes. The use of the generalized Eluder dimension and covering numbers enables analysis in non-tabular, high-capacity function approximation regimes encompassing nonlinear architectures (e.g., deep RL). The value gap separation condition is comparatively weaker than source/target domain shift assumptions studied in [qu2024hybrid] and more reflective of the practical "partial coverage" observed in empirical offline-to-online transfer literature [nakamoto2023cal, yu2023actor].

Empirical Validation

The algorithm is empirically instantiated on MuJoCo AntMaze benchmarks, initializing from Cal-QL pretrained ensembles and updating via ensemble standard deviation-based interval estimation. O2O-LSVI consistently matches or outperforms strong offline RL and adaptation baselines (CQL, IQL, Cal-QL) across varied data quality and coverage settings. This empirically confirms the regime identified in the theory: when the offline value function is accurate on a large but not full domain, O2O-LSVI efficiently exploits it while retaining exploration guarantees elsewhere.

Implications and Future Directions

The presented lower bound closes previous theoretical-experimental gaps, refuting the notion that proximity of the offline-pretrained value function to optimality alone implies accelerated adaptation. Structural conditions—such as partial high-confidence regions or explicit value gaps—are shown to be necessary for provably faster adaptation.

This framework lays the foundation for sharper characterization of transferability in RL with high-capacity function classes, and for new algorithms that judiciously identify and exploit reliable knowledge derived from heterogeneous or limited offline data sources. Future work may focus on variance-aware algorithms, adaptive misspecification detection, and broader application to model-based RL under dynamic shifts.

Conclusion

This paper offers a theoretically principled and practically validated offline-to-online RL algorithm, O2O-LSVI, provably leveraging accurate offline value functions when and only when such functions satisfy structural gap and coverage conditions. It establishes tight limits on transfer efficiency, advances the state of theoretical RL with general function approximation, and provides actionable algorithmic guidance for settings with partial offline coverage and online adaptation requirements (2604.13966).