SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation

Abstract: LLM agents are increasingly used to translate natural language into 3D scenes in a procedural way, but existing systems focus on static output. Dynamic 4D scenes from text alone, in which liquids flow, particles emit, rigid bodies cascade, and articulated mechanisms move, remain largely unexplored despite their value as editable content and as physics-grounded training data for video generation and embodied AI. Two challenges set the dynamic case apart from static text-to-scene work: an agent must jointly coordinate spatial layout, multiple physics solvers, temporal sequencing, camera, and lighting in a single coherent scene, and verifying motion correctness from rendered video is fundamentally harder than judging a single image. We present SimWorlds: a multi-agent framework that produces dynamic, editable 4D scenes from text, with Blender-specific procedural knowledge, a planner-coder-reviewer workflow driving a fixed ordered sequence of construction stages, a layered scene protocol enforced by a deterministic verifier, and a runtime-state inspection tool suite that catches mechanism failures the rendered image cannot reveal. We also introduce 4DBuildBench, a benchmark for assessing both visual fidelity and physical consistency of the procedural dynamic 3D scenes generated from text prompts. Experiments show that SimWorlds outperforms prior dynamic Blender generation baselines.

• The paper introduces a multi-agent system that automatically generates dynamic, mechanism-grounded 3D Blender scenes from text prompts.
• The methodology integrates planner, coder, and reviewer agents with deterministic verification to ensure simulation correctness across staged pipeline steps.
• Benchmarking shows significantly higher mechanism and structural pass rates versus previous methods, validating its robust, physics-driven content generation.

SimWorlds: Multi-Agent LLM-Driven Generation of Dynamic, Mechanism-Grounded 3D Scenes

Overview and Motivation

SimWorlds ("SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation" (2607.01766)) introduces a paradigm shift in text-to-3D generation by automating the generation of dynamic, controllable, and editable 3D Blender scenes directly from natural language. Unlike previous LLM-agent pipelines—where the focus has primarily been on static 3D scenes—SimWorlds addresses the complexity of synthesizing dynamic, physics-driven 4D content (i.e., scenes with temporally coherent motions, physical interactions, and simulation states that are verifiable and editable).

The core innovation is a multi-agent framework (planner, coder, reviewer) orchestrated via a staged, pipeline architecture, integrating deterministic verification mechanisms and direct engine state inspection. This work targets crucial downstream applications, including physically-grounded synthetic data generation, simulation environments for embodied AI, and content creation workflows where downstream editability, physical parameterization, and compositionality are essential.

Figure 1: SimWorlds turns text into dynamic, editable 3D Blender scenes. Given a natural-language prompt, a planner, coder, and reviewer cooperate to emit a .blend whose geometry, materials, lighting, camera, and motion all remain controllable for downstream editing and reuse.

System Architecture and Methodology

SimWorlds formalizes the dynamic text-to-scene problem as plan-grounded, mechanism-aware program synthesis. The system orchestrates scene construction via three specialized LLM agents, each with distinct roles and tool affordances:

• Planner: Converts input text into an explicit scene plan, enumerating objects, spatial relationships, physical roles, animation and simulation breakdowns, and downstream rendering intent.
• Coder: Decomposes the scene plan into pipeline stages, emitting Blender Python scripts (using bpy) to manipulate geometry, assign materials, add simulators (rigid/soft bodies, fluids, particles, cloth), and configure temporal sequencing.
• Reviewer: Evaluates per-stage outputs for visual plausibility and, crucially, mechanism correctness by inspecting Blender's internal state and runtime data, rather than only visually rendered outputs.

Distinctively, the pipeline is fixed and stratified: modeling, UV, texturing, deformation setup, motion, camera, lighting, rendering. Each stage includes a deterministic verifier based on a declarative scene protocol. Verification is not limited to visual QA, but reads Blender engine state for key properties (e.g., existence of baked simulation caches, correct modifier stacks, active solver configurations).

Figure 2: Pipeline overview. The planner compiles the prompt into a single scene plan; construction then proceeds through a fixed stage sequence, each stage running the coder, a deterministic verifier (scene protocol), and a reviewer, with failed checks triggering bounded local retries. Once all stages close, the scene is rendered as a frame-sequence video.

A layered "scene protocol" complements Blender's object/collection model, ensuring correct object-level grouping, explicit encoding of spatial and relational constraints (contacts, co-movement, containment), and grounding in the expected physical simulation mechanisms. This protocol is enforced via a suite of structural, geometric, and semantic checks at each pipeline stage.

A powerful tool suite, including direct scene state queries, multi-angle previews, and structured runtime readouts, empowers agents to introspect and correct for non-obvious mechanism failures—e.g., distinguishing physically solved animation from keyframed "fakes".

Benchmarking and Evaluation: 4DBuildBench

SimWorlds is evaluated on 4DBuildBench, a rigorously constructed 50-scene benchmark spanning five Blender solver mechanisms—cloth, fluid, rigid body, particle system, and soft body—across a tailored difficulty axis (from single-actor to cross-category interaction), and a static scene category for control. Prompts demand both visual accuracy and correct mechanism instantiation.

Critically, SimWorlds is assessed using two orthogonal metrics:

• Mechanism Pass Rate (MPR): Deterministic audit of .blend files for correct use and configuration of physics solvers, objects, and relations (e.g., confirms if a flowing liquid is simulated, not keyframed).
• Structural Pass Rate (SPR): Measures geometric and relational correctness, e.g., proper physical contacts, spatial arrangements.
• VLM-Based Visual Judge: Itemized evaluation using GPT-5.5, assigning binary scores to object, relation, and action correctness in the rendered video frames, but unable to detect mechanism fakes.

This dual-track evaluation is crucial: Visual-only judgment is trivially spoofed by keyframed or static mesh manipulation, while engine-state audits guarantee physical editability and mechanism correctness.

Comparative and Ablation Results

SimWorlds decisively outperforms VIGA (the previous SOTA dynamic Blender generator) on all mechanism-grounded metrics. Key strong claims, supported by quantitative data:

• MPR: SimWorlds achieves 0.87 vs. VIGA's 0.67 overall; the difference is even more pronounced on fluid scenes (0.95 vs. 0.56).
• SPR: 0.89 vs. 0.70 overall, reflecting improved spatial and structural scene correctness.
• VLM Score: Comparable (0.82 vs. 0.78), confirming that prior benchmarks overweight visual plausibility at the expense of simulation validity.

Ablations clarify the contribution of the deterministic verifier and staged construction pipeline. Removal of the verifier causes substantial SPR drop; removing staged construction degrades VLM scores, illustrating complementary roles in enforcing structural and aesthetic/visual polish.

Figure 3: Qualitative comparison across four dynamic sequences: (a) a domino cascade in a child's bedroom; (b) a paper airplane landing on a cluttered desk; (c) a wind-blown curtain billowing over a desk; (d) a glass hourglass pouring sand from its upper to its lower bulb.

Qualitative results reinforce quantitative findings: Only SimWorlds systematically realizes the correct mechanism for non-trivial dynamic prompts (e.g., actual domino cascades via rigid-body simulation, proper fluid particles for sand flows), while VIGA frequently admits visually plausible but non-editable or non-simulated "fakes".

Practical and Theoretical Implications

SimWorlds sets a new benchmark for dynamic, mechanism-aware text-to-3D scene generation. Its architecture ensures that generated content is not merely a rendered artifact, but a procedural, editable asset with all simulation states and compositional assets accessible for downstream manipulation (e.g., AI training, further editing, simulation-based rendering, or system identification for physics studies).

Practically, SimWorlds narrows the reality gap for synthetic data pipelines in computer vision, robotics, and graphics, providing assets with faithful mechanism semantics and temporal structure. The staged pipeline and deterministic protocol abstraction offer a reusable scaffold for other LLM-driven procedural generation settings, whether in varying simulation engines or content domains.

Theoretically, the work demonstrates the necessity of external, environment-grounded verification in long-horizon LLM-agent systems for non-trivial procedural tasks. The failure of visual-only agents to recognize mechanism-level errors underlines critical limits in self-critique and calls for more sophisticated grounding protocols and introspection tools.

Limitations and Future Directions

SimWorlds remains text-only; cross-modal conditioning (e.g., with reference images or sketches) represents a clear path for increased control. Perceptual and compositional judgments (e.g., aesthetics, scene "sensibility") depend on VLM guidance and are less robust than deterministic checks. Richer compositional editing and support for additional simulation engines or asset formats may enhance generality.

Extending this framework to learn planning heuristics from execution trajectories, integrating constraint optimization for scene feasibility, or supporting interactive co-creation with humans are promising research trajectories. Pluggable, open protocol verification will likely become standard in generative agent pipelines.

Conclusion

SimWorlds establishes a pipeline for mechanism-aware, editable, and visually faithful text-to-4D scene synthesis in Blender. By introducing a multi-agent staged pipeline with strong, engine-grounded verification and a comprehensive public benchmark, it raises expectations for both content authenticity and procedural editability in scene generation research. The explicit separation of mechanism and perceptual evaluation in benchmarking sets a precedent for future work at the intersection of AI agents and procedural graphics.