Intelligent Go-Explore (IGE) in Reinforcement Learning

• Intelligent Go-Explore (IGE) is a family of adaptive reinforcement learning algorithms that automate archive management and exploration using learned state representations and foundation models.
• Adaptive post-exploration and time-myopic embeddings enable efficient coverage and robust handling of sparse or deceptive rewards in complex, high-dimensional environments.
• FM-driven variants leverage pretrained models for semantic novelty, significantly reducing cell conflicts and overcoming detachment in exploration archives.

Intelligent Go-^^^^1^^^^ (IGE) is a family of algorithms that generalize and extend the Go-Explore paradigm for hard-exploration reinforcement learning by automating or learning the decision rules for archiving, returning to, and outwardly exploring from states. Whereas classic Go-Explore leveraged hand-designed cell representations, archive sampling heuristics, and ad hoc novelty metrics, recent IGE variants utilize adaptive post-exploration modules, self-supervised learned state representations, and, most recently, foundation models (FMs) that encode implicit notions of semantic novelty and promise. This approach provides robust, scalable, and semantically meaningful exploration in high-dimensional and/or complex environments with sparse or deceptive rewards (Ecoffet et al., 2020, Yang et al., 2022, Höftmann et al., 2023, Lu et al., 2024).

1. Foundations of Go-Explore and the Motivation for IGE

Go-Explore was initially proposed to address limitations of RL methods in environments where rewards are sparse or exploration requires many temporally extended, precise decisions. The core mechanism involves maintaining an explicit archive $\mathcal{A}$ of previously discovered states (typically organized via a cell discretization), repeatedly returning to promising archived states ("Go"), and launching new exploratory episodes from them ("Explore"). This avoids both detachment (forgetting how to return to promising regions) and derailment (destructive, unintentional exploration paths) (Ecoffet et al., 2020).

In its original formulation, Go-Explore required the user to provide all critical components: archive cell representation, archive update rules, state selection weights, and criteria for adding new states. This manual tuning limited generality and scalability, motivating the need for intelligent, automated alternatives (Lu et al., 2024).

2. Adaptive Post-Exploration: Formal Methods and Empirical Results

An early form of IGE extended policy-based Go-Explore by integrating adaptive post-exploration. The method, framed as an Intrinsically Motivated Goal Exploration Process (IMGEP), introduces a phase of random exploration appended after reaching each goal state: for every reached goal $g$, the agent executes $n_{\rm pe}$ random actions, incentivizing exploration at the frontier of known space (Yang et al., 2022).

Adaptive control is achieved via:

• When to post-explore: set the probability of post-exploration as $p_{\rm pe}(g) = [1/n(g)]^{\beta}$, with $n(g)$ the visit count to $g$ and $\beta$ a scaling parameter. This prioritizes rarely visited ("frontier") goals.
• How long to post-explore: the number of exploration steps is set proportionally to the episode length required to reach $g$, $n_{\rm pe} = p_{\rm len} n_{\rm ep}$, with $p_{\rm len} \in (0,1]$.

Empirical evaluations on MiniGrid environments (FourRooms, LavaCrossing, LavaGap) showed static post-exploration can increase coverage and learning speed, but adaptive versions (tuning $\beta$ and $p_{\rm len}$) further accelerate and focus exploration. For instance, $\beta=0.01$ yielded complete coverage $\sim10\%$–$15\%$ faster than static strategies, and proportional post-exploration reduced the total number of noisy steps required for full coverage (Yang et al., 2022).

3. Learned State Representations: Time-Myopic Embedding Principles

Classic Go-Explore's reliance on downscaled pixel hashings or handcrafted cell features poses significant drawbacks in high-dimensional, ambiguous, or stochastic domains. The time-myopic Go-Explore variant (an editor's term for this IGE instantiation) addresses this by learning a continuous state representation that clusters temporally proximate observations and configures novelty as time-based embedding distance (Höftmann et al., 2023).

The core components include:

• A siamese convolutional encoder $\Phi_\theta$ mapping $k$-step observation pairs to a $D$-dimensional latent space.
• A time predictor $\Psi_\theta$ estimating normalized temporal distances between embeddings:

$$\Psi_\theta(z_t, z_{t+k}) = 1 - \exp\bigl(-\,\max\{\,f_\theta(z_{t+k}-z_t),\,0\}\bigr)\; \in [0,1]$$

• The loss is the MSE between $\Psi_\theta(z_t, z_{t+k})$ and $\min(k/L,1)$, where $L$ is a maximum time horizon.

Novelty is judged as $$d_{\min}(z_K) = \min_{C\in\mathcal{A}} \Psi_\theta(z_C, z_K)$$, and new states are added to the archive if $d_{\min}(z_K) > T_d$. The archive update is strictly insertion-only, ensuring monotonic coverage and eliminating cell conflicts (no two semantically different states unintentionally collapse) (Höftmann et al., 2023).

Experiments on Atari games (Montezuma's Revenge, Gravitar, Frostbite) showed that the learned representation achieves comparable or improved coverage and, notably, maintains much smaller and less redundant archives (e.g., 566 vs. 2800 cells at similar scores in Montezuma's Revenge). The time-myopic design inherently resolves detachment and avoids archive overwriting.

4. Automated, Foundation Model-Driven IGE

A recent advance is the direct integration of giant pretrained foundation models to remove the need for human-designed cell representations and exploration heuristics. In this FM-driven IGE, state selection, action choice, and archive updating are handled via LLM or multimodal FM queries that draw on learned human-like notions of novelty and promise (Lu et al., 2024).

Key mechanisms:

• FM-guided state selection: The archive of descriptions is given to the FM, which selects the next state for exploration based on implicit "interestingness" or progress potential.
• FM-guided exploration: The FM proposes actions (zero-shot, chain-of-thought, etc.), leveraging prior context and reasoning to maximize discovery.
• FM-based archive filtering: Candidate states from exploration are presented to the FM, which determines—possibly non-verbally or via scalar scores—which are "interestingly new" and should be archived. This enables recognition of serendipitous discoveries, i.e., valuable but unanticipated states.

This approach, evaluated on language-based and hybrid environments (Game of 24, BabyAI-Text, TextWorld), achieves success rates that strongly exceed RL and classic graph search baselines as well as SOTA FM agents like Reflexion, especially in long-horizon and semantically complex scenarios (Lu et al., 2024). Ablation studies confirm substantial performance drops when any FM-guided component is removed.

5. Addressing Detachment, Cell Conflicts, and Robust Exploration

Standard Go-Explore architectures suffer from detachment (the archive "forgets" promising regions due to cell overwrites) and cell conflicts (different meaningful states are accidentally merged). The insertion-only, continuous embedding strategy in IGE directly prevents these failures: no cells are overwritten, and the criterion for novelty is a learned time-based separation, reducing both redundancy and semantic collapse (Höftmann et al., 2023).

When foundation models are used, the archive filtering process can reason over explicit representations (descriptions, embeddings) to further separate truly distinct discoveries, and adapt exploration in response to previously unseen environmental semantics (Lu et al., 2024).

Adaptive post-exploration focuses computational effort at the boundaries of explored regions, and the combination of hindsight relabeling and replay drives sample efficiency in both sparse-reward and dense-reward regimes (Yang et al., 2022).

6. Limitations and Future Directions

While IGE offers significant advances, key limitations remain:

• FM Dependency: Performance is sensitive to FM quality; replacing GPT-4 with GPT-3.5 leads to dramatic decreases in success (e.g., from 92% to 0% on the Cooking Game) (Lu et al., 2024).
• Scalability: FM-guided archive lookups across thousands of states are compute- and token-intensive. Query cost in learned-state IGE remains $O(|\mathcal{A}|)$ due to exhaustive distance calculations (Höftmann et al., 2023, Lu et al., 2024).
• Modality coverage: Existing FM-driven IGE variants have not been extensively demonstrated in vision-only or continuous-control robotics domains. Extensions will require robust multimodal FM architectures (Lu et al., 2024).
• Granularity and long-range novelty: The learned time-myopic embeddings saturate for pairs exceeding the trained time window $L$; thus, discrimination among extremely distant states is reduced (Höftmann et al., 2023).
• Sample efficiency gap: In some environments (e.g., Frostbite), classic Go-Explore remains more sample-efficient than time-myopic IGE under the reported protocol (Höftmann et al., 2023).

Proposed future directions include retrieval-augmented FM queries (for sublinear archive filtering), end-to-end joint policy-representation learning, scalable abstraction in continuous/3D spaces, and combining IGE-generated experience with offline RL policy distillation (Höftmann et al., 2023, Lu et al., 2024).

7. Summary Table: Key IGE Mechanisms Across Variants

Variant	Archive Representation	Exploration Heuristic	Novelty/Filtering	Core Differentiator
Classic Go-Explore	Handcrafted cell mapping	Frontier or count-based	Hash or rule-based insert	Fully manual, domain-dependent
Adaptive PE IGE	Observed states	Adaptive post-explore	Visit-count, proportional	Adaptive, sample-efficient exploration
Time-Myopic IGE	Siamese latent embedding	Weighted by reward/visits	Learned time-distance	Continuous, detachment-free
FM-driven IGE	Descriptions, embeddings	FM-guided action/return	FM-determined "interestingness"	Semantic, serendipity-aware

This organizational progression from manual rules to adaptive modules to learned and semantic representations underlies the development of Intelligent Go-Explore, a general class of exploration algorithms defining a new research frontier in scalable, sample-efficient, and semantically robust exploration for RL and related AI tasks (Ecoffet et al., 2020, Yang et al., 2022, Höftmann et al., 2023, Lu et al., 2024).