Pure Exploitation Learning (PEL)

• PEL is a decision-making algorithm that maximizes expected rewards using a greedy strategy without explicit exploration incentives.
• It recovers near-optimal exploratory behavior when recurring environmental structure, sufficient agent memory, and long-horizon credit assignment are present.
• Theoretical guarantees and empirical results demonstrate that PEL’s regret bounds in exogenous MDPs rival those of traditional exploration-based methods.

Pure Exploitation Learning (PEL) is a class of decision-making algorithms in which the agent pursues a strictly greedy (exploitation-only) objective, eschewing any explicit exploration incentives such as stochastic action selection, optimism, or intrinsic reward bonuses. Recent research demonstrates that under the right structural conditions, PEL can recover optimal or near-optimal exploratory behavior as a direct consequence of reward maximization, and admits strong theoretical guarantees in settings where environment randomness is exogenous or environmental and agent memory structures are recurrent (Rentschler et al., 2 Aug 2025, Liang et al., 28 Jan 2026).

1. Formal Foundations of Pure Exploitation Learning

PEL formalizes the agent's objective as maximizing expected discounted return without explicit exploration incentives. For a (possibly partially observable) MDP with agent history $h_t$ at time $t$, discounted by $\gamma_{\text{step}}\in[0,1]$ and with fixed transition $P$ and reward $r$, the PEL objective is: $$\max_{\pi}\; \mathbb{E}_{s_0\sim\rho,\,a_t\sim\pi(\cdot\mid h_t),\,s_{t+1}\sim P(\cdot\mid s_t,a_t)}\Bigl[\sum_{t=0}^{T-1} \gamma_{\rm step}^t\,r(s_t,a_t)\Bigr]$$ The value function is propagated through the greedy Bellman update,

$$Q^*(h_t,a) = \mathbb{E}\bigl[r(h_t,a)+\gamma_{\rm step}\max_{a'}Q^*(h_{t+1},a')\bigr]$$

The deployed policy is deterministic and greedy: $$\pi^*(a\mid h_t) = \mathbf{1}\!\Bigl[a = \arg\max_{a'}\,Q^*(h_t,a')\Bigr]$$ Exogenous MDPs (Exo-MDPs) introduce further structure: each state decomposes as $s_h = (x_h, \xi_h)$, where $\xi_h$ is exogenous and evolves independently of agent actions. Exploitation-only methods in Exo-MDPs estimate exogenous transitions and rewards empirically, then apply dynamic programming or value regression to construct the greedy policy (Liang et al., 28 Jan 2026).

2. Necessary and Sufficient Conditions for Emergent Exploration

Experiments and ablations establish three principal conditions for emergent exploration under PEL in meta-RL:

• Recurring Environmental Structure: Environment parameters (reward, transition, observation) persist across episodes in a task block of mean length $n$ (geometric distribution). This enables accumulation and exploitation of information across episodes; for $n=1$, systematic exploration disappears.
• Agent Memory: The policy leverages a context-dependent memory (typically a causal transformer with context window $X$) storing recent agent experience. Sufficiently large $X$ is critical; below a threshold ($X\approx 64$ bandit, $128$–$1024$ gridworlds), emergent exploration vanishes.
• Long-Horizon Credit Assignment: When credit is propagated across episodes using an episode-level discount $\gamma_{\rm episode} > 0$ alongside $\gamma_{\rm step}$, agents are incentivized to perform actions that benefit future as well as present outcomes. However, in stateless bandit tasks, emergent exploration persists even with $\gamma_{\rm episode}=0$ due to implicit “pseudo-Thompson Sampling” from the transformer architecture (Rentschler et al., 2 Aug 2025).

In Exo-MDPs, the decoupling of exogenous and endogenous transitions ensures that exploiting policy can learn exogenous dynamics independently of agent actions, undergirding strong regret guarantees for PEL (Liang et al., 28 Jan 2026).

3. Theoretical Guarantees and Analytical Tools

PEL achieves rigorous regret bounds in Exo-MDPs, both tabular and with linear function approximation. In the tabular setting, cumulative regret after $K$ episodes is: $$\mathrm{Regret}(K) = \widetilde O\left(H^2|Ξ|\sqrt{K}\right)$$ For large or continuous endogenous state/action spaces with linear features of dimension $d$, the LSVI-PE algorithm satisfies: $$\mathrm{Regret}(K) = O\left(\left(\frac{N}{\lambda_0} + d\right)\,|Ξ|\,H\,\sqrt{K\,\ln \tfrac{1}{\delta}}\right)$$ and simplifies to $\widetilde O(d\,|Ξ|\,H\,\sqrt{K})$ in well-conditioned cases, independent of $|X|$ or $|A|$ (Liang et al., 28 Jan 2026).

Analysis leverages two main tools:

• Counterfactual Trajectories: Enables unbiased evaluation of optimal-policys’ value by replaying the same exogenous trace with the optimal policy on endogenous dynamics.
• Bellman-Closed Feature Transport: Under post-decision linearity, feature maps transport linearly with bounded operator norm, ensuring stability and convergence of regression updates.

No explicit optimism or forced exploration is needed; regret bounds are established using concentration inequalities and the independence structure of exogenous variables.

4. Empirical Evaluations and Experimental Protocols

PEL has been evaluated in both meta-RL (repeated bandits/gridworlds) and exogenous MDP (synthetic control/resource management) settings.

Meta-RL (PEL in Repeat-Structured Tasks) (Rentschler et al., 2 Aug 2025):

• Bandit Task: $n=30$ episodes per block, $X=1024$, $\gamma_{\rm episode}=0.9$.
  • Meta-RL agent attains normalized reward $0.70$ (Thompson Sampling $0.61$; $\epsilon$-greedy $0.50$).
  • Performance collapses as $n\to1$ or $X$ decreases below threshold ($X=32$: $-0.052$).
• Gridworld Task: $n=30$, $X=1024$, $\gamma_{\rm episode}=0.9$.
  • Achieves normalized reward $\approx0.67$ (oracle $1.0$, random $0$), state visitation heatmaps evidence broad initial exploration converging to efficient exploitation.

Exo-MDPs (LSVI-PE, PTO) (Liang et al., 28 Jan 2026):

• Tabular Exo-MDP: $X=[5]$, $\Xi=[5]$, $A=[3]$, $H=5$, $K=250$.
  • PTO achieves lower cumulative regret than exploration-based baselines (PTO-Opt, PTO-Lite).
• Continuous Resource Control: Inventory/storage, exogenous price states, $H\le10$, $K=100$.
  • LSVI-PE outperforms UCB-augmented and subsampled baselines across all settings, validating the sufficiency of exploitation-only learning when exogenous dynamics are reused.

5. Emergent Pseudo-Thompson Sampling and Model Capacity Effects

In cases lacking explicit long-horizon credit assignment ($\gamma_{\rm episode}=0$), transformer-based meta-RL agents can mimic Thompson Sampling—termed "pseudo-Thompson Sampling" (“pseudo-TS”). This arises because:

• When the Bellman target degenerates to immediate reward, traditional neural networks regress to the mean.
• Causal transformers fine-tuned via LoRA adapters exhibit in-context stochasticity: randomized, context-dependent outputs sample different return hypotheses across runs.
• The greedy action is chosen for the context-specific $Q$ estimate, and the stochastic nature of in-context learning is equivalent to sampling in a manner analogous to Thompson Sampling.

This phenomenon critically depends on (i) stateless task structure, (ii) large context window $X$, and (iii) sufficient model capacity (Rentschler et al., 2 Aug 2025).

6. Implications, Limitations, and Future Directions

PEL establishes that, in the presence of recurring environmental structure, sufficient agent memory, and (for temporally extended tasks) long-horizon credit assignment, pure reward maximization recovers sophisticated exploratory behavior—even outperforming explicit exploration strategies in certain domains.

• Implications for Algorithm Design: PEL simplifies design by obviating the need for exploration hyperparameters. It provides a unified objective aligning exploration with exploitation, with direct applicability to meta-RL and operations research problems where exogenous stochasticity dominates (Rentschler et al., 2 Aug 2025, Liang et al., 28 Jan 2026).
• Theoretical Impact: PEL overcomes the longstanding view that explicit exploration is fundamental by providing regret bounds that hold without optimism or exploration-driving mechanisms.
• Limitations: When environmental structure is non-recurrent, credit assignment across episodes is challenging, or high-dimensional continuous domains with sparse rewards are involved, pure exploitation methods may be insufficient, and explicit exploration incentives or distributional value learning become necessary.
• Open Problems: Further research is needed to delineate the precise boundaries and robustness of emergent exploration, especially in continuous-state, nonlinear, and high-dimensional regimes.

7. Table: Comparative Summary of PEL Conditions and Outcomes

Environment Structure	Agent Memory	Emergent Exploration	Performance
Recurrence ($n\gg1$), $X>X^*$	Present	Yes	Near-optimal
Recurrence ($n=1$)	Any	No	Random/Myopic
Any	$X<X^*$	No	Random/Myopic
Exo-MDP (exogenous, any $X$)	N/A	Yes	Regret-optimal

$X^*$: memory threshold determined empirically per task class.

---

For comprehensive exposition and experimental methodology, see (Rentschler et al., 2 Aug 2025, Liang et al., 28 Jan 2026).