Experiential Reinforcement Learning

Abstract: Reinforcement learning has become the central approach for LMs to learn from environmental reward or feedback. In practice, the environmental feedback is usually sparse and delayed. Learning from such signals is challenging, as LMs must implicitly infer how observed failures should translate into behavioral changes for future iterations. We introduce Experiential Reinforcement Learning (ERL), a training paradigm that embeds an explicit experience-reflection-consolidation loop into the reinforcement learning process. Given a task, the model generates an initial attempt, receives environmental feedback, and produces a reflection that guides a refined second attempt, whose success is reinforced and internalized into the base policy. This process converts feedback into structured behavioral revision, improving exploration and stabilizing optimization while preserving gains at deployment without additional inference cost. Across sparse-reward control environments and agentic reasoning benchmarks, ERL consistently improves learning efficiency and final performance over strong reinforcement learning baselines, achieving gains of up to +81% in complex multi-step environments and up to +11% in tool-using reasoning tasks. These results suggest that integrating explicit self-reflection into policy training provides a practical mechanism for transforming feedback into durable behavioral improvement.

• The paper introduces ERL, a novel reinforcement learning method that integrates explicit self-reflection to revise agent behavior.
• It employs a two-step process where an initial attempt is followed by reflective revision, improving rewards in tasks like Sokoban and FrozenLake.
• Experimental results demonstrate that ERL efficiently transforms feedback into durable policy improvements, outperforming standard RL approaches.

Experiential Reinforcement Learning: Explicit Self-Reflective Revision for Language Agent Optimization

Introduction and Motivation

The paper "Experiential Reinforcement Learning" (2602.13949) introduces ERL, a novel paradigm for optimizing LLMs deployed as decision-making agents via reinforcement learning augmented with explicit self-reflection and behavioral consolidation. The standard approach for RL in LLMs relies solely on scalar rewards, either directly from environment signals or outcome-based verification, propagating corrections through policy gradients. While effective in interactive settings, this design suffers from slow, unstable learning, especially under sparse and delayed rewards, due to implicit credit assignment and undirected exploration.

ERL addresses these deficiencies by structuring learning into an explicit cycle: the model generates an initial attempt, receives environmental feedback, self-reflects to produce a revisionary prompt, attempts the task anew conditioned on this reflection, and internalizes successes into the base policy via distillation. This process enables targeted behavioral correction within each trajectory and produces lasting policy improvement not dependent on feedback or reflection at inference time.

Figure 1: ERL requires agents to verbally self-reflect on experience and outcomes, internalizing reflections to revise and consolidate future behavior.

Algorithmic Architecture and Training Dynamics

ERL augments standard RL with an experience–reflection–consolidation loop, operationalized as follows:

• For each input task $x$, the agent first generates $y^{(1)}$ and observes feedback $f^{(1)}$, reward $r^{(1)}$.
• If $r^{(1)} < \tau$ (gated mechanism), the agent produces a structured reflection $\Delta$ based on $(x, y^{(1)}, f^{(1)}, r^{(1)}, m)$, where $m$ is a memory of previously successful reflections.
• Conditioned on $\Delta$, the agent generates a refined second attempt $y^{(2)}$, receiving new feedback $f^{(2)}$, reward $r^{(2)}$.
• Both attempts and reflections are optimized via a reward-driven policy gradient objective, with distillation loss $\mathcal{L}_{\text{distill}}$ consolidating successful second attempts into the base policy (input $x$ only).

The architecture ensures that corrections are not just locally effective but persistently encoded in the deployable model. The cross-episode reflection memory $m$ further stabilizes self-reflection and enables cumulative refinement.

Figure 2: Overview of ERL: initial attempt, self-reflection, revised attempt, RL optimization, and self-distillation for internalization.

Comparison to Standard RL Approaches

Standard RLVR (Reinforcement Learning with Verifiable Reward) agents update only via terminal reward signals, driving routine trial-and-error exploration. While flexible, RLVR does not provide explicit corrective structure, leading to repeated re-exploration and inefficient learning, especially under sparse or delayed reward. ERL introduces verbal, self-administered revision steps within the same training episode, guided by feedback, yielding persistent improvements and enhancing exploration efficiency. This difference is conceptualized in:

Figure 3: RLVR relies on scalar rewards; ERL explicitly employs self-reflection-driven revision and consolidation to enable durable policy improvement.

Experimental Evaluation

ERL and RLVR were evaluated on three agentic reasoning tasks: FrozenLake, Sokoban, and HotpotQA, using Olmo-3-7B-Instruct and Qwen3-4B-Instruct-2507. These environments feature sparse terminal rewards and unknown dynamics, requiring agentic reasoning and strategic exploration.

Key empirical findings:

• In Sokoban, ERL led to a reward increase from 0.06 (RLVR) to 0.87 (ERL) with Qwen3 and from 0.04 to 0.20 with Olmo3, indicating substantial gains in challenging control tasks requiring long-horizon planning.
• In FrozenLake, improvements ranged up to +27% absolute reward, demonstrating ERL's advantage in environments with unknown symbol semantics and reasoning requirements.
• For HotpotQA, a tool-augmented multi-hop QA task, ERL improved reward by up to +11%, showing reliable but less pronounced gains due to denser reward and reduced reliance on latent dynamics.

  Figure 4: ERL achieves higher reward and faster improvement than RLVR across model architectures and tasks.

  

  Figure 5: Final evaluation reward comparison on all tasks, confirming ERL consistently outperforms RLVR.

  

  Figure 6: ERL post-reflection trajectories consistently achieve higher reward than RLVR and ERL pre-reflection.

Mechanistic Analysis and Ablation Studies

Analysis of reward trajectories pre- and post-reflection highlights ERL's within-episode correction mechanism: immediate reward increases are observed after reflection-guided retry. Ablation studies further demonstrate the importance of both structured self-reflection and cross-episode memory:

• Disabling memory slows convergence; removing reflection leads to significant reward loss.
• In some stochastic settings, persistent memory can propagate inaccurate reflections, but overall, ERL with memory is strongly preferred.
• Structured reflection produces actionable behavioral corrections; memory enables their propagation and stabilization.

Practical and Theoretical Implications

Practically, ERL enhances training efficiency, stability, and final policy quality for LLM agents operating in environments with sparse rewards and complex dynamics. Structuring RL as experience–reflection–consolidation transforms raw feedback into actionable corrections, reducing dependence on undirected exploration and facilitating durable performance improvements.

Theoretically, ERL aligns with contemporary views advocating for experiential learning in AI, where explicit credit assignment and continual self-improvement are treated as core primitives. By formalizing reflection within the RL trajectory and consolidating improvements into a deployable policy absent reflection at inference, ERL bridges inference-time reflection approaches with agent-centric RL optimization.

These insights suggest new directions for scaling RL in LLMs: integrating verbal reasoning modules, structured self-reflection, and memory systems for persistent adaptation, potentially enabling continual lifelong learning and rapid recovery in novel environments.

Conclusion

Experiential Reinforcement Learning introduces a principled approach to embedding explicit experience–reflection–consolidation dynamics within RL optimization for LLM agents. Empirical results across diverse tasks and architectures demonstrate substantial improvements in learning efficiency and policy quality relative to standard RL baselines. The mechanism of reflection-driven revision and selective internalization provides a practical blueprint for transforming environmental feedback into persistent behavioral correction. Future developments in AI will likely extend ERL's core primitives—reflection, memory, and consolidation—to more complex, multi-agent, and continual learning settings, driving the next advances in adaptive agentic intelligence.