One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration

Abstract: Symbolic world modeling requires inferring and representing an environment's transitional dynamics as an executable program. Prior work has focused on largely deterministic environments with abundant interaction data, simple mechanics, and human guidance. We address a more realistic and challenging setting, learning in a complex, stochastic environment where the agent has only "one life" to explore a hostile environment without human guidance. We introduce OneLife, a framework that models world dynamics through conditionally-activated programmatic laws within a probabilistic programming framework. Each law operates through a precondition-effect structure, activating in relevant world states. This creates a dynamic computation graph that routes inference and optimization only through relevant laws, avoiding scaling challenges when all laws contribute to predictions about a complex, hierarchical state, and enabling the learning of stochastic dynamics even with sparse rule activation. To evaluate our approach under these demanding constraints, we introduce a new evaluation protocol that measures (a) state ranking, the ability to distinguish plausible future states from implausible ones, and (b) state fidelity, the ability to generate future states that closely resemble reality. We develop and evaluate our framework on Crafter-OO, our reimplementation of the Crafter environment that exposes a structured, object-oriented symbolic state and a pure transition function that operates on that state alone. OneLife can successfully learn key environment dynamics from minimal, unguided interaction, outperforming a strong baseline on 16 out of 23 scenarios tested. We also test OneLife's planning ability, with simulated rollouts successfully identifying superior strategies. Our work establishes a foundation for autonomously constructing programmatic world models of unknown, complex environments.

• The paper introduces OneLife, a novel framework that learns modular symbolic world models from a single, unguided exploration in stochastic environments.
• It leverages a mixture of modular laws with precondition-effect structures and unsupervised LLM-based law synthesis, optimizing parameters via L-BFGS for efficient inference.
• The framework demonstrates superior predictive and planning performance, outperforming baselines in state ranking and fidelity across 23 diverse game mechanics.

Inferring Symbolic World Models from Unguided Exploration in Stochastic Environments: The OneLife Framework

Introduction and Motivation

The paper presents OneLife, a framework for learning symbolic world models in complex, stochastic environments from a single unguided episode, without access to environment-specific rewards or goals. This addresses a critical challenge in world modeling: inferring the underlying transition dynamics of an environment when interaction is severely limited and the agent must operate without human guidance. The approach is motivated by the need for interpretable, modular, and verifiable models that can generalize across diverse mechanics and stochastic behaviors, as found in open-world games like Crafter.

Figure 1: OneLife synthesizes world laws from a single unguided episode, modeling the world as a mixture of code-based laws with precondition-effect structure and inferring parameters to best explain observed dynamics.

Framework Overview: Mixture of Modular Laws

OneLife models the environment's transition function as a probabilistic mixture of modular laws, each implemented as a program with a precondition-effect structure. The environment is represented as a pure transition function $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$, where $\Delta(\mathcal{S})$ denotes distributions over the state space. Each law $L_i$ is defined by a precondition $c_i(s, a)$ and an effect $e_i(s, a)$, allowing the law to remain silent on irrelevant aspects and only predict attributes it governs. This modularity enables efficient credit assignment and targeted learning, as only active laws are updated during inference.

Figure 2: The inference process routes updates only through laws implicated by observed changes, yielding a dynamic computation graph for efficient learning.

The world model is formalized as a probabilistic program generating distributions over observables, with the predictive distribution for each observable $o$ given by a weighted product of predictions from active laws:

$$p(o=v | s, a; \boldsymbol{\theta}) \propto \prod_{i \in \mathcal{I}_o(s, a)} \phi_i(o=v | s, a)^{\theta_i}$$

where $\mathcal{I}_o(s, a)$ is the set of active laws relevant to $o$.

Law Synthesis and Parameter Inference

Law synthesis is performed by prompting a LLM to propose atomic laws for each observed transition, decomposing complex events into fine-grained rules. This process is unsupervised and leverages the object-oriented structure of the environment state, avoiding domain-specific hand-engineering. The parameter inference stage fits the weights $\boldsymbol{\theta}$ by maximizing the log-likelihood of observed transitions, updating only the laws responsible for observed changes. The optimization is performed using L-BFGS, exploiting the sparsity of the dynamic computation graph.

Evaluation Protocols and Metrics

The evaluation protocol is designed to rigorously assess both the discriminative and generative capabilities of the learned world model:

• State Ranking: Measures the ability to assign higher probability to the true next state over distractors generated by mutators that violate environment rules. Metrics include Rank@1 and Mean Reciprocal Rank (MRR).
• State Fidelity: Quantifies the edit distance between the predicted and true next state, using raw and normalized JSON Patch operations.

  Figure 3: Two evaluation metric categories: state ranking (discriminative) and state fidelity (generative), with distractors generated by mutators.

Evaluation is performed on Crafter-OO, a reimplementation of Crafter with a structured, object-oriented state and pure transition function, enabling precise programmatic manipulation and comprehensive scenario coverage.

Experimental Results

OneLife demonstrates superior predictive judgment compared to strong baselines, including PoE-World, across 23 core game mechanics. The framework achieves an MRR of 0.479 and Rank@1 of 18.7%, outperforming PoE-World by 7.9 percentage points and 0.128 MRR, respectively. State fidelity remains competitive, with normalized edit distance comparable to baselines.

Figure 4: Per-scenario state ranking performance (MRR) of OneLife vs. PoE-World, grouped by core game mechanic.

Fine-grained analysis shows that OneLife outperforms PoE-World in 16 out of 23 scenarios, indicating robust generalization across diverse mechanics rather than overfitting to simple cases.

Planning in Imagination

The learned world model supports forward simulation for planning, enabling agents to evaluate alternative strategies without real-world interaction. In three scenarios (combat, resource collection, and tool use), simulated rollouts within OneLife's world model correctly identify superior plans, matching the ground-truth environment's ranking of strategies.

Figure 5: Example of plan execution in OneLife's world model for the Stone Miner scenario, correctly simulating causal dependencies for effective and ineffective plans.

This demonstrates that OneLife captures the causal structure necessary for goal-directed reasoning and can simulate long action sequences ($>30$ steps) with high fidelity.

Implementation Considerations

• Computational Requirements: The modular law structure and dynamic computation graph enable efficient inference, as only implicated laws are updated per transition. The use of L-BFGS for parameter optimization is tractable due to the sparsity of updates.
• Scalability: The framework is designed to scale to environments with large, hierarchical state spaces and many interacting mechanics, as only relevant laws are activated per transition.
• Limitations: Exploration remains a bottleneck; unguided LLM-based policies can fail to discover rare mechanics or progress through complex tech trees. The approach assumes access to a structured, object-oriented state, which may require significant engineering in some environments.
• Deployment: The learned symbolic world model is interpretable and editable, supporting downstream applications in planning, verification, and transfer to related domains.

Implications and Future Directions

The OneLife framework establishes a foundation for autonomous reverse engineering of environment dynamics from minimal, unguided interaction. Its modular, interpretable structure facilitates integration with symbolic planners, verification tools, and programmatic RL agents. Future work may focus on improving exploration policies, automating law synthesis for environments with less structured state, and extending the approach to multi-agent or partially observable domains. The evaluation protocol and Crafter-OO environment provide a robust testbed for benchmarking symbolic world modeling methods in realistic, stochastic settings.

Conclusion

OneLife advances symbolic world modeling by enabling agents to infer modular, probabilistic laws from a single unguided episode in complex, stochastic environments. The framework achieves strong discriminative and generative performance, supports planning in imagination, and generalizes across diverse mechanics. Its design principles and evaluation methodology set a new standard for interpretable, data-efficient world modeling in open-ended domains.