Experiential-Symbolic Task Framework

• Experiential-symbolic tasks are defined by combining direct interaction with symbolic abstractions to overcome limitations of traditional deep RL.
• They employ memory-based methods to encode and transfer causal constraints via dependency graphs, enabling rapid zero-shot and few-shot generalization.
• Dual-control policies blend exploration for uncertainty reduction with exploitation strategies, significantly improving sample efficiency in both simulated and real environments.

An experiential-symbolic task constitutes a class of problems in which agents must integrate experiential learning—direct interaction with and observation of the environment—with explicit, discrete symbolic representations in order to efficiently acquire, transfer, and execute complex behavior. These tasks are characterized by the presence of discontinuous state-action affordances, causal dependencies among action components, and the necessity for rapid knowledge transfer across task instances or variants. Rather than relying solely on continuous value functions or model-free reinforcement learning, experiential-symbolic frameworks exploit high-level discrete abstractions, symbolic constraints, and structured memory to overcome the limitations of traditional deep learning in domains with sharp causal structure or combinatorial generalization requirements (Verghese et al., 2023).

1. Formal Structure of Experiential-Symbolic Tasks

Let $x \in X \subset \mathbb{R}^N$ denote the continuous state space of the environment, and let $s \in S$ be a symbolic abstraction obtained via a mapping from $x$ to discrete symbolic features—for example, $s_i \in \{0,1\}$ indicating unlocked/locked status of $N$ components. Actions $a \in \{1, \dots, N\}$ typically correspond to toggling or manipulating individual components. Crucially, the feasible action set $A(s)$ may vary discontinuously with the symbolic state: mechanical or geometric interlocks (e.g., "door cannot be opened until handle is turned") induce discontinuous affordances captured by symbolic constraints.

These constraints are naturally encoded via a directed dependency graph $G = (V, E)$, with vertices representing components and edges $e^{ij}$ capturing "component $i$ locks $j$" relationships. The movability of a component $j$ at state $s$ is given by:

$C(s,j) = \neg \bigvee_{i \neq j} s_i \land e^{ij}$

where $C(s, j)$ being true indicates that $j$ is not locked by any $i$.

Experiential-symbolic tasks thus combine:

• A high-dimensional continuous or hybrid dynamical system
• Discrete symbolic state abstractions, typically with strong causal and temporal constraints
• Discontinuous, state-dependent action affordances and preconditions
• Temporal/causal ordering among actions (e.g., via a partial order or explicit memory)

2. Memory-Based Learning and Symbolic Constraint Transfer

Experiential-symbolic learning departs from end-to-end value function approximation by employing a memory system that stores discrete relational knowledge gained from experience. After solving a task instance, the agent identifies the strongly supported edges (constraints) in $G$—those with high posterior probability $P(e^{ij}=1) \approx 1$—and encapsulates them along with task-specific features such as the component class and relative pose.

Concretely:

• Each dependency edge $(i,j)$ is stored as a tuple $(c_i, c_j, \Delta p, P(e^{ij}=1))$, where $c_i, c_j$ are component classes and $\Delta p$ encodes geometric relation.
• When facing a new variant, initialization of $P(e^{i'j'})$ employs $K$-nearest neighbor retrieval in the $(c_{i'}, c_{j'}, \Delta p)$ space and sets the prior based on the mean retrieved values, introducing small noise to maintain exploration entropy.

This symbolic memory enables:

• Rapid zero-shot or few-shot generalization, as constraints are transferred across geometric isomorphs
• Avoidance of catastrophic forgetting and weight overwriting, circumventing issues inherent to fine-tuning in neural value-based RL
• Instance-based recall and updating of both discrete and geometric task features

3. Dual-Control Policy: Action Selection and Uncertainty Reduction

Action selection in experiential-symbolic frameworks leverages a dual-objective Q-value structure:

• $Q_\mathrm{info}(s_t, a)$ quantifies the expected reduction in uncertainty over $G$ (i.e., expected KL divergence between current and next belief states), promoting exploration of yet-unsolved structure.
• $Q_\mathrm{exploit}(s_t, a)$ estimates the maximal probability of reaching the goal under the current constraint graph, typically computed via shortest-path search in the symbolic transition graph given current $P(E_t)$.

The two Q-functions are blended by the current entropy $h$ of the graph belief state:

$$Q(s_t, a) = \frac{h}{h_\mathrm{max}} Q_\mathrm{info}(s_t, a) + \left(1-\frac{h}{h_\mathrm{max}}\right) Q_\mathrm{exploit}(s_t, a)$$

The selected action $a_t$ maximizes $Q(s_t, a)$.

This blended policy guarantees efficient exploration during initial constraint discovery and aggressive exploitation as the agent converges on correct symbolic rules.

4. Empirical Evaluation and Comparative Efficiency

Experimental results across synthetic and real domains demonstrate the distinct sample efficiency advantages of experiential-symbolic task algorithms:

• In simulated locking-puzzle suites (5 heterogeneous components, 9 layouts), memory-based methods reach $\sim$70% success after 9 episodes and 100% after about 30 training episodes, while model-based and model-free RL require an order of magnitude more episodes to achieve similar success rates.
• In real robotic disassembly (8 components), memory-based approaches converge in 20 episodes (17 actions), compared to >150 episodes for dense-reward deep RL baselines.
• In sim-to-real transfer settings, memory-based agents solve unseen puzzles in 6.6 actions after 30 training episodes, outperforming model-based RL (21.2 actions) and matching model-free RL only when the latter is allowed $\sim$700 episodes.

Order-of-magnitude reductions in required environmental interaction stem from explicit knowledge transfer via symbolic-relational memory and the nonparametric, experience-driven instantiation of priors for new task variants.

5. Mechanisms Enabling Rapid Transfer and Generalization

The fundamental enablers of experiential-symbolic transfer are:

• Explicit encoding of causal constraints as discrete dependency graphs that generalize exactly under geometric morphisms, avoiding the smoothing effect of neural representations on discontinuities.
• Instancing of rules by component class and relative pose, enabling combinatorial re-use not accessible to weight-based function approximation.
• Memory-based policy initialization that obviates online relearning or weight adaptation, as new tasks are addressed via fast KNN in memory, sidestepping gradient-based optimization entirely.

For example, once the rule "Slide locks Door if $\Delta p$ in $R^\mathrm{lock}$" is established, any novel (Slide$'$, Door$''$) pair with $\Delta p' \in R^\mathrm{lock}$ immediately inherits this knowledge, suppressing futile exploration—unlike deep RL, which would require repeated credit assignment to shape the value function in this new region.

6. Context, Extensions, and Theoretical Significance

Experiential-symbolic task frameworks address environments with sharply discontinuous or combinatorially structured affordances—a regime poorly served by standard continuous-value RL. This methodology decisively supports systems where:

• The state-action relevance is strongly state-dependent and masking is essential
• Symbolic knowledge acquired in one context can apply with minimal adaptation in others, via geometric or class-structural invariance
• Autonomous systems must exhibit rapid adaptability and robustness to unseen variations without full retraining

These frameworks are foundational for lifelong learning agents, zero-shot transfer in robotics, automated planning in reconfigurable environments, and other domains demanding an overview of symbolic reasoning and interaction-driven learning (Verghese et al., 2023).

---

References:

(Verghese et al., 2023) Using Memory-Based Learning to Solve Tasks with State-Action Constraints