Text-to-3D Scene Generation

• Text-to-3D scene generation is a process that transforms textual descriptions into spatially structured 3D scenes using advanced NLP, computer vision, and generative modeling techniques.
• It employs diverse representations—scene graphs, implicit fields, and explicit primitives—to ensure semantic fidelity, geometric consistency, and physical plausibility.
• Diffusion models and physics-based optimization techniques are key to achieving photorealistic detail, stable object placement, and interactive scene editing.

Text-to-3D scene generation refers to the automatic synthesis of structured, spatially coherent three-dimensional scenes directly from textual prompts. This research field integrates advances in natural language processing, computer vision, and 3D generative modeling to enable applications such as virtual world creation, embodied agent simulation, film production, design, and gaming. Recent advances leverage vision-LLMs (VLMs), LLMs, diffusion-based generative models, and scene representations such as scene graphs, 3D Gaussian splatting, radiance fields, and explicit meshes (Ruiz et al., 18 Nov 2025, Zhang et al., 2023, Zhang et al., 2023, Li et al., 18 Jul 2025, Xiong et al., 7 Apr 2025, Zhang et al., 2 Apr 2025, Chu et al., 27 Jan 2026, Yang et al., 2024, Zhou et al., 4 Feb 2025, Chen et al., 2023, Li et al., 2024, Li et al., 2024, Zhou et al., 2024, Lin et al., 26 Nov 2025, Ling et al., 5 May 2025, Chang et al., 2015, Chang et al., 2017, Zhou et al., 2024). The domain has converged around several architectural paradigms and evaluation protocols for semantic fidelity, geometric consistency, and physical plausibility.

1. Representations for 3D Scene Synthesis

Text-to-3D scene generation relies on diverse scene representations. Early systems constructed 3D scenes by selecting assets from model databases and arranging them according to probabilistic spatial priors or rule-based templates (Chang et al., 2017, Chang et al., 2015). Modern generative systems adopt parameterized 3D representations:

• Scene Graphs: Graphs in which nodes represent objects (with class, location, orientation, and attributes) and edges encode spatial or semantic relations. For example, GeoSceneGraph represents a scene as $G = (V, E, X)$ where each node aggregates class embedding, shape code, bounding box, and centroid, and edges are implicit (fully connected), supporting E(3)-equivariant updates (Ruiz et al., 18 Nov 2025).
• Implicit Fields: Neural Radiance Fields (NeRFs) and signed distance functions parameterized by MLPs facilitate continuous geometry and realistic novel-view rendering (Zhang et al., 2023, Zhang et al., 2023). These are often used as environment backbones or for implicit object/background representation.
• Explicit Primitives: 3D Gaussian splatting is now standard for representing both environments and objects, offering explicit, differentiable, and memory-efficient volumetric rendering. Each Gaussian is specified by mean, covariance, opacity, and color coefficients (Li et al., 18 Jul 2025, Xiong et al., 7 Apr 2025, Li et al., 2024, Yang et al., 2024, Chu et al., 27 Jan 2026, Zhou et al., 2024, Lin et al., 26 Nov 2025).
• Hybrid Models: Some pipelines use explicit (e.g., DMTet or mesh) representations for salient objects and implicit fields for backgrounds, integrating crisp object-level control with flexible scene modeling (Zhang et al., 2023, Kang et al., 26 Sep 2025).

2. Text and Semantic Conditioning Schemes

A core challenge is grounding textual semantics into spatially distributed 3D entities and relations:

• Language Encoding: Text inputs are consistently mapped into high-dimensional embeddings via CLIP-based encoders or direct LLMs. These embeddings condition generative modules in several ways: concatenation with node/features (Ruiz et al., 18 Nov 2025), cross-attention into neural layers (Ruiz et al., 18 Nov 2025, Zhang et al., 2023), or direct scene-graph construction by a LLM (Li et al., 18 Jul 2025, Zhou et al., 2024, Zhang et al., 2023, Zhou et al., 4 Feb 2025, Ling et al., 5 May 2025).
• Scene Graph Extraction: Systems such as GeoSceneGraph, GALA3D, and LayoutDreamer parse scene graphs using LLMs or lightweight NLP pipelines, associating objects, spatial relations, and coarse bounding volumes (Ruiz et al., 18 Nov 2025, Zhou et al., 2024, Zhou et al., 4 Feb 2025). These graphs serve as priors for layout and composition modules.
• Multi-stage Planning: DreamScene and Scenethesis employ LLM-based planners to draft coarse layouts, which guide downstream optimization and iterative refinement (Li et al., 18 Jul 2025, Ling et al., 5 May 2025).
• Conditioned Diffusion Guidance: Conditioning on text may be injected as an “edge message” within graph neural network diffusion modules (Ruiz et al., 18 Nov 2025), as a control branch in 2D/3D diffusion (Chen et al., 2023, Zhang et al., 2023), or as layout maps and structured input to ControlNets (Zhou et al., 2024).

3. Scene Layout, Geometric Reasoning, and Physical Plausibility

Determining plausible and controllable 3D object placement is central:

• Combinatorial Layout: Particle Swarm Optimization (PSO) is used for sampling object poses maximizing CLIP-text alignment (SceneWiz3D) (Zhang et al., 2023). Graph-based placement algorithms traverse scene graphs, enforcing region anchors and spatial relations (DreamScene) (Li et al., 18 Jul 2025).
• Physics-Guided and Physically-Aware Placement: Multiple systems inject energy-based or physically plausible objectives—
  • PAT3D uses a differentiable rigid-body simulator enforcing static equilibrium and intersection-free placement, with a semantics loss enforcing alignment to the scene tree (Lin et al., 26 Nov 2025).
  • LayoutDreamer and PhiP-G introduce explicit physical energies (gravity, penetration, contact, anchor) and iterative adjustments for collision avoidance, stability, and alignment, guided by scene graphs or visual agents (Zhou et al., 4 Feb 2025, Lin et al., 26 Nov 2025).
  • Scenethesis applies signed distance field–based losses for collision elimination, object stability, and coherence (Ling et al., 5 May 2025).
• Iterative Layout Correction: Visual and language agents analyze current scenes and recommend layout corrections for gaps, overlaps, and “floating” objects (PhiP-G, Scenethesis) (Zhou et al., 4 Feb 2025, Ling et al., 5 May 2025).
• Adaptive Path Planning: RoamScene3D uses scene-graph reasoning to plan camera trajectories that explore salient objects, ensuring visibility and refined inpainting (Chu et al., 27 Jan 2026).

4. Diffusion-Based 3D Generation and Optimization

Diffusion models and their variants underpin modern text-to-3D pipelines:

• Score Distillation Sampling (SDS): 3D scene representations are optimized such that rendered images at sampled camera poses minimize the SDS objective against 2D diffusion priors, using classifier-free or control-guided noise predictions (Li et al., 2024, Li et al., 18 Jul 2025, Chen et al., 2023, Zhou et al., 2024, Zhang et al., 2023).
• Multi-Timestep and Multi-View Sampling: Formation Pattern Sampling (Darknet: FPS) samples diffusion times over a shrinking window, capturing both semantic and geometric cues, with final stages concentrated on reconstructive generation for photorealistic detail (Li et al., 2024, Li et al., 18 Jul 2025).
• Feed-Forward 3D Diffusion: Models like Prometheus and Director3D extend latent diffusion to directly produce pixel-aligned or world-aligned 3D Gaussians in a feed-forward (“seconds-level”) fashion, utilizing joint RGB-D latent spaces, multi-view denoising, and hybrid classifier-free guidance schemes (Yang et al., 2024, Li et al., 2024).
• Joint Objects-and-Scene Optimization: Instance-level or compositional SDS steps on objects are followed by scene-level diffusion (often with conditioned ControlNets), aligning global interactions and style (Zhou et al., 2024, Zhang et al., 2023, Kang et al., 26 Sep 2025).
• Inpainting and Augmentation for Consistency: Drift in low-coverage or occluded regions is mitigated by panoramic diffusion priors, motion-injected inpainting, or explicit panorama reguidance (SceneWiz3D, RoamScene3D, DreamScene360, PanoDreamer, WorldPrompter) (Zhang et al., 2023, Chu et al., 27 Jan 2026, Zhou et al., 2024, Xiong et al., 7 Apr 2025, Zhang et al., 2 Apr 2025).

5. Novel Scene Editing, Control, and Interactivity

Recent pipelines support scene editing, interactive adjustment, and fine-grained control:

• Model-Driven Editing: Scene graphs and layout descriptors permit object relocation, property modification, and re-optimization (DreamScene, GALA3D, LayoutDreamer, Scenethesis) (Li et al., 18 Jul 2025, Zhou et al., 2024, Zhou et al., 4 Feb 2025, Ling et al., 5 May 2025).
• Sketch and Multi-modal Control: Control3D demonstrates direct sketch-conditioned 3D scene generation, using a ControlNet branch and explicit sketch-consistency loss over rendered views to guide volumetric NeRFs (Chen et al., 2023).
• 4D Dynamics and Motion Synthesis: DreamScene and Drag4D enable user- or LLM-defined 4D scene evolution: animated trajectories are realized by re-rendering dynamic objects, or via part-aware video diffusion in joint 3D layouts (Li et al., 18 Jul 2025, Kang et al., 26 Sep 2025).
• Traversability and Navigation: WorldPrompter and RoamScene3D produce truly traversable, walkable 3D worlds by generating panoramic videos aligned to the text then reconstructing them into globally consistent Gaussian fields permitting real-time navigation (Zhang et al., 2 Apr 2025, Chu et al., 27 Jan 2026).

6. Evaluation Protocols and Comparative Metrics

Evaluation is multifaceted, using both automated quantitative indices and human studies:

• Text–3D Fidelity: CLIP-Score (cosine similarity between rendered images and prompt), Q-Align metrics, and user studies (1–10 or 1–5 scales) are widely used (Zhou et al., 2024, Zhou et al., 4 Feb 2025, Li et al., 18 Jul 2025, Li et al., 2024, Chu et al., 27 Jan 2026).
• Perceptual Quality: NIQE, BRISQUE, and FID (especially FID-CLIP and FID over disparities) are employed for image/geometry quality (Li et al., 18 Jul 2025, Zhang et al., 2023, Xiong et al., 7 Apr 2025, Yang et al., 2024, Li et al., 2024, Chu et al., 27 Jan 2026).
• Geometric and Physical Plausibility: Collision rates (object/scene, Col-O/Col-S), instability (Inst-O/Inst-S), and displacement under simulated gravity are reported (Lin et al., 26 Nov 2025, Ling et al., 5 May 2025). Intersection-free and stability metrics distinguish methods with explicit physic simulation.
• Coverage and Consistency: Scene traversability (WorldPrompter), multi-view consistency (R-Precision, CLIP-AP, and alignment losses), and coverage metrics track whether scenes maintain semantic and geometric coherence over wide trajectories (Zhang et al., 2 Apr 2025, Zhang et al., 2023, Yang et al., 2024).

7. Limitations and Future Directions

Current limitations include:

• Long-horizon/Scale: Scalability to large, highly cluttered, or open outdoor scenes is reduced due to memory or depth estimation constraints (Chu et al., 27 Jan 2026, Zhou et al., 2024, Zhang et al., 2 Apr 2025).
• Physics and Non-Rigidity: Most methods treat all entities as rigid; handling non-rigid, articulated, or deformable objects is rare (Lin et al., 26 Nov 2025).
• Prompt Generalization: Failure cases arise for prompts with complex spatial prepositions, rare relations, or unseen object classes; addressing these may require LLM finetuning or multimodal fusion (Zhang et al., 2023, Ruiz et al., 18 Nov 2025).
• Interactive and Real-Time Editing: While compositional pipelines support local re-optimization, fully end-to-end and differentiable layout networks for real-time scene editing are still in development (Zhang et al., 2023, Zhou et al., 2024).

Future research directions include integration of stronger LLMs for richer and more fine-grained scene graph extraction (Chu et al., 27 Jan 2026, Li et al., 18 Jul 2025), end-to-end scene planning to generation networks, extension to dynamic and outdoor environments, richer material/lighting modeling, and the fusion with embodied agent and robotics pipelines (Lin et al., 26 Nov 2025, Ling et al., 5 May 2025, Ruiz et al., 18 Nov 2025, Zhou et al., 2024).

---

References:

(Ruiz et al., 18 Nov 2025, Zhang et al., 2023, Zhang et al., 2023, Li et al., 18 Jul 2025, Xiong et al., 7 Apr 2025, Zhang et al., 2 Apr 2025, Chu et al., 27 Jan 2026, Yang et al., 2024, Zhou et al., 4 Feb 2025, Chen et al., 2023, Li et al., 2024, Li et al., 2024, Zhou et al., 2024, Lin et al., 26 Nov 2025, Ling et al., 5 May 2025, Chang et al., 2015, Chang et al., 2017, Zhou et al., 2024).