SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness

Abstract: Recent unified image generation models have achieved remarkable success by employing MLLMs for semantic understanding and diffusion backbones for image generation. However, these models remain fundamentally limited in spatially-aware tasks due to a lack of intrinsic spatial understanding and the absence of explicit geometric guidance during generation. In this paper, we propose SpatialFusion, a novel framework that internalizes 3D geometric awareness into unified image generation models. Specifically, we first employ a Mixture-of-Transformers (MoT) architecture to augment the MLLM with a parallel spatial transformer to enhance 3D geometric modeling capability. By sharing self-attention with the MLLM, the spatial transformer learns to derive metric-depth maps of target images from rich semantic contexts. These explicit geometric scaffolds are then injected into the diffusion backbone through a specialized depth adapter, providing precise spatial constraints for spatially-coherent image generation. Through a progressive two-stage training strategy, SpatialFusion significantly enhances performance on spatially-aware benchmarks, notably outperforming leading models such as GPT-4o. Additionally, it achieves generalized performance gains across both text-to-image generation and image editing scenarios, all while maintaining negligible inference overhead.

• The paper introduces SpatialFusion, a unified architecture that couples semantic parsing with explicit 3D geometric reasoning using a Mixture-of-Transformers paradigm.
• The methodology employs a two-stage training paradigm, including geometric-aware pretraining and joint optimization, to internalize spatial priors within diffusion models.
• Empirical evaluations show SpatialFusion outperforms traditional models by 10–30 points on spatial tasks, enhancing image compositionality and enabling precise instruction-based editing.

SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness

Motivation and Problem Statement

Unified image generation models leverage Multimodal LLMs (MLLMs) combined with diffusion backbones to generate semantically faithful images from complex prompts. Although this paradigm achieves state-of-the-art results in general visual realism and prompt alignment, it is fundamentally limited in spatially-conditioned and geometric reasoning tasks. Systematic evaluation reveals that current MLLMs produce feature representations that lack intrinsic 3D geometry. These geometry-deficient embeddings, when propagated to diffusion-based generators, yield outputs that are spatially incoherent, lack proper object poses, and fail in metric-sensitive compositions (Figure 1).

Figure 1: Existing MLLMs and diffusion models lack explicit spatial priors, resulting in geometry-deficient feature representations and spatially inconsistent image synthesis; SpatialFusion addresses these deficiencies.

Prior approaches attempting to address spatial awareness in generation typically inject external geometric cues (e.g., bounding boxes, segmentation maps, or externally computed depth signals) into the generative pipeline. However, these methods require additional inputs unavailable in pure text-to-image scenarios, or act only as conditional executors without achieving intrinsic 3D understanding. Thus, engineering a unified generative model with internalized, end-to-end 3D geometric awareness constitutes an open challenge.

Method: SpatialFusion Architecture

SpatialFusion proposes a unified solution by explicitly coupling geometric reasoning pathways with standard semantic pipelines (Figure 2). The architecture integrates:

• An MLLM for semantic context parsing
• A parallel Spatial Transformer (ST) for explicit 3D geometric representation
• A variational autoencoder (VAE) for latent compression
• A diffusion transformer (DiT) backbone for high-fidelity synthesis

  Figure 2: SpatialFusion uses MoT-based semantics-guided geometric derivation and injects explicit 3D scaffolds into the generative diffusion model to enforce spatial constraints.

A Mixture-of-Transformers (MoT) paradigm is employed: the ST shares self-attention with the MLLM but maintains independent projection matrices to build specialized geometric reasoning pathways. Geometric queries are initialized and augmented with robust DINOv2 features. The ST derives dense metric-depth maps, which serve as explicit structural scaffolds, by integrating semantic context throughout its layers. The resulting 3D scaffolds are processed by a lightweight depth adapter and injected as additive residuals into the diffusion latents prior to each denoising step. This mechanism achieves geometry-constrained image generation with minimal architectural disruptions and parameter overhead.

Training Paradigm

SpatialFusion leverages a two-stage curriculum:

1. Geometric-Aware Pretraining: The ST and geometric tokens are pretrained under the supervision of a frozen 3D foundation model (VGGT), using pseudo-ground-truth metric-depth derived from static targets. Semantic cues are regressed onto explicit depth predictions, creating proactive geometric scaffolds from semantics alone.
2. Geometry-Guided Joint Training: The pretrained ST is incorporated in the end-to-end pipeline, and both it and the diffusion backbone are jointly optimized on a large-scale, geometry-augmented dataset (including rich spatial and editing tasks). A joint loss objective balances geometric regression and visual synthesis fidelity, with the curriculum transitioning from broad multimodal alignment to high-precision spatial fine-tuning.

The resulting model internalizes geometric priors, supporting spatially-aware T2I and instruction-driven editing tasks with negligible inference overhead.

Experimental Analysis

Spatial Awareness on the GenSpace Benchmark

SpatialFusion achieves dominant scores across all categories (Spatial Pose, Relation, Measurement) in GenSpace for both text-to-image and editing, surpassing MLLM-diffusion baselines and outpacing competitive unified models such as GPT-4o. For example, in camera/object pose and spatial relation tasks, SpatialFusion improves accuracy margins by 10–30 absolute points over the next-best models. Gains are more pronounced in spatial relation subdomains demanding allocentric or intrinsic understanding, with robustness in metric-constrained image synthesis.

Generalization to Standard Benchmarks

On comprehensive benchmarks including GenEval, T2I-CompBench++, and DPG-Bench, SpatialFusion does not trade off performance in color, texture, or compositional reasoning. Instead, intrinsic 3D awareness generalizes to overall generative quality, achieving state-of-the-art or near-best performance across all measured axes. Particularly, improvements in spatial reasoning metrics are not coupled with regressions in linguistic flexibility or fine-grained visual attributes.

Figure 3: Qualitative results demonstrating metric-depth map guided synthesis, showing spatially coherent image generation consistent with geometric scaffolds.

Instruction-Based Editing

On structure-sensitive instruction-based editing (ImgEdit-Bench, GEdit-Bench-EN), explicit geometric integration results in substantial improvements for tasks such as object addition, replacement, and removal, especially under geometric constraints. Editing operations blend objects at correct perspectives, and inpainting strictly adheres to plausible spatial completions.

Qualitative Comparison

Figure 4: SpatialFusion produces geometrically and semantically compliant images, outperforming prior models in spatially-aware generation across diverse prompts.

Visual comparisons against state-of-the-art generative and unified models highlight that SpatialFusion preserves object orientation, relative layouts, and metric scaling with high fidelity (Figure 4). Competing models frequently exhibit spatially implausible or inconsistent arrangements, especially under complex or compositionally rich instructions.

In-Depth Ablation and Analysis

Ablation studies reveal that:

• Uniform shared attention sampling between the ST and MLLM optimally grounds geometric tokens across both low- and high-level features.
• Additive fusion of depth features into latents yields better spatial control than concatenative methods, avoiding distributional disruptions to pretrained priors.
• The geometric regression weight $\lambda$ must be tuned to avoid overfitting on geometry at the cost of synthesis fidelity; $\lambda=0.5$ achieves best trade-off.

Computational analysis confirms that the additional inference latency is minimal (<3% per image); the total parameter increase is <20%, preserving deployment feasibility.

Implications and Future Directions

SpatialFusion demonstrates that internalizing explicit geometric reasoning within a generative model not only solves spatially-aware synthesis but also yields broad improvements in image compositionality, editing, and visual-semantic integration. The architecture and training paradigm validate the hypothesis that a unified transformer framework—leveraging both semantic and geometric pathways—enables cross-modal pretraining transfer, supporting the integration of strong spatial priors even in the absence of explicit 3D scene inputs.

Key theoretical implications:

• Semantic-geometric decoupling within a MoT structure facilitates internalization of 3D priors with minimal interference with existing semantic representations.
• Dense 3D supervision (pseudo-depth from foundation models) can be distilled into generative backbones, enabling zero-shot generalization on spatial tasks.

Practically, this enables production-grade, instruction-driven image generation and editing with spatial control, useful in AR/VR, robotics simulation, and controllable media synthesis contexts.

Future research directions may explore:

• Generalization to unified video generation with 3D spatiotemporal awareness
• Incorporation of richer geometric modalities (e.g., surface normals, multi-view consistency)
• Transfer to open-world scene understanding and simulation

Conclusion

SpatialFusion establishes a framework for end-to-end geometry-aware image generation by integrating MoT-based semantics-guided 3D derivation and diffusion with explicit spatial priors. Empirical results confirm that this approach enables robust, scalable, and controllable synthesis, outperforming prior paradigms on both spatial and standard generative benchmarks, and pointing to promising directions for unified vision-language-geometry pretraining (2604.26341).