M3: 3D-Spatial MultiModal Memory

Abstract: We present 3D Spatial MultiModal Memory (M3), a multimodal memory system designed to retain information about medium-sized static scenes through video sources for visual perception. By integrating 3D Gaussian Splatting techniques with foundation models, M3 builds a multimodal memory capable of rendering feature representations across granularities, encompassing a wide range of knowledge. In our exploration, we identify two key challenges in previous works on feature splatting: (1) computational constraints in storing high-dimensional features for each Gaussian primitive, and (2) misalignment or information loss between distilled features and foundation model features. To address these challenges, we propose M3 with key components of principal scene components and Gaussian memory attention, enabling efficient training and inference. To validate M3, we conduct comprehensive quantitative evaluations of feature similarity and downstream tasks, as well as qualitative visualizations to highlight the pixel trace of Gaussian memory attention. Our approach encompasses a diverse range of foundation models, including vision-LLMs (VLMs), perception models, and large multimodal and LLMs (LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy M3's feature field in indoor scenes on a quadruped robot. Notably, we claim that M3 is the first work to address the core compression challenges in 3D feature distillation.

• The paper presents M3, a multimodal memory system that integrates 3D Gaussian splatting with foundation models to capture spatial and semantic scene details.
• It employs Principal Scene Components and Gaussian Memory Attention to compress high-dimensional features and enable efficient feature retrieval.
• Experimental results across datasets show superior performance in metrics like PSNR and SSIM compared to existing methods.

M3: 3D-Spatial MultiModal Memory

The paper "M3: 3D-Spatial MultiModal Memory", identified by arXiv ID (2503.16413), introduces a novel multimodal memory system known as 3D Spatial MultiModal Memory (M3). This system is designed for the efficient retention and semantic representation of medium-scale static scenes through video sources, integrating 3D Gaussian Splatting techniques with various foundation models to store information in a structured Gaussian format. This essay provides a detailed examination of the proposed methodology, related works, and experimental results.

Introduction

The ability to store and retrieve spatial and semantic information about surrounding environments is a hallmark of human cognition that many previous models have struggled to emulate. Existing approaches like NeRF (Pham et al., 2021) and 3D Gaussian Splatting (3DGS) (Zhao et al., 2024) achieve pixel-level scene representation but fall short of capturing semantic aspects. Addressing these gaps, M3 utilizes 3D Gaussian splatting and foundation models to build a comprehensive multimodal memory system capable of rendering high-fidelity feature representations over multiple scales of granularity. This paper highlights two primary challenges in previous works: computational constraints in managing high-dimensional features and potential feature misalignments. M3 mitigates these through innovations such as principal scene components and Gaussian memory attention, which enable efficient training and inference with minimal information loss.

Figure 1: Our proposed MultiModal Memory integrates Gaussian splatting with foundation models to efficiently store multimodal memory in a Gaussian structure. The feature maps rendered by our approach exhibit high fidelity, preserving the strong expressive capabilities of the foundation models.

The M3 architecture consists of two major components: principal scene components (PSC) to compress high-dimensional features, and Gaussian memory attention for efficient feature retrieval. Figure 2 visually depicts the comprehensive pipeline of the M3 system.

Figure 2: A scene ($V$) composed of structure ($S$) and knowledge ($I$) rendered using Gaussian splatting integrated with foundation models.

Methodology

The M3 system employs the concept of Visual Granularity (VG) and Knowledge Space (KS) to organize and store multimodal information. Visual granularity represents the clustering pixel scope of an image from fine details to large-scale elements, designed to mirror human perception at multiple granularities. Knowledge Space, on the other hand, refers to the diverse aspects of knowledge, such as visual alignment, semantics, and reasoning, captured via various foundation models. M3 encodes this knowledge and structure using Gaussian splatting, constructing a full-stack multimodal memory of a static scene.

Figure 2: A scene ($V$) is composed of both structure ($S$) and knowledge ($I$). To model these, we leverage multiple foundation models to extract multi-granularity scene knowledge and employ 3D Gaussian splatting to represent the spatial structure. By combining these techniques, we construct a spatial multimodal memory (M3), which enables downstream applications such as retrieval, captioning, and grounding.

A key feature of M3 is its utilization of 3D Gaussian splatting for scene reconstruction, preserving both geometric and semantic features (Figure 2). High-dimensional 2D feature maps from diverse foundation models are compressed into what are referred to as Principal Scene Components (PSC), which are then retrieved using low-dimensional Principal Queries (Q) from Gaussian primitives. A Gaussian Memory Attention mechanism links these queries to the PCS and employs them in composition with other foundational model components, creating a comprehensive multimodal memory of the scene.

Figure 2: A scene ($V$) is composed of both structure ($S$) and knowledge ($I$). To model these, we leverage multiple foundation models to extract multi-granularity scene knowledge, and employ 3D Gaussian splatting to represent the spatial structure.

Experiments

Quantitative Results

Experiments were performed using a series of established datasets including Mip-NeRF360, Tanks and Temples, DeepBlending, and a custom-developed M3-Robot dataset. The study juxtaposes M3 against recent distillers like F-3DGS (Zhao et al., 2024) and F-Splat (Qiu et al., 2024). The results from low-level assessments, along with metrics such as PSNR, SSIM, L2, and Cosine metrics, reveal superior feature memorization for M3 across a variety of foundation models. The turn to Gaussian Memory Attention provides distinct alignment gains over existing distillation techniques, apparent in the improved performance metrics compared to F-3DGS and F-Splat2 (Table 1).

Qualitative Results

The visualization of M3 outputs demonstrates high fidelity in preserving expressive spatial structures alongside diverse semantic information across datasets such as Garden, Playroom, Drjohnson, and Table-top (Figure 2).

Figure 3: Qualitative results across datasets using M3. The figure showcases the consistent performance of the M3\ across various datasets (Garden, Playroom, Drjohnson, Table-top).

This consistent performance underscores M3's capability to maintain both structural and semantic elements across varying granularities of scene detail, notably performing well on complex arrangements and objects with overlapping attributes.

Implications and Future Work

The research presents significant advancements in the creation of a 3D-Spatial MultiModal Memory that achieves multimodal integration in scene representation, bridging the gap between spatial granularity and knowledge richness. The proposed Gaussian Memory Attention mechanism effectively enhances the alignment of features with knowledge spaces. The experimental deployment in real-world scenarios, such as on a quadruped robot platform, showcases M3's robustness and versatility for practical applications. Future developments may involve enhancing the reasoning capabilities by incorporating a dedicated reasoning module to further extend M3's applications in complex, dynamic environments, as well as better reduce feature compression limitations inherent in current approaches.

Conclusion

The study presents a spatial multimodal memory system, M3, that strategically integrates 3D Gaussian splatting with foundation models to enhance the representation of both structure and semantic knowledge of scenes. This integration offers a significant improvement over previous feature splatting methodologies, particularly in its ability to retain high-dimensional feature representations while enabling efficient training. The research demonstrates the model's adaptability across multiple datasets and various practical applications, including robotic implementation. Future adaptations could further enhance the reasoning capacities of M3\, elevating its utility in evolving AI systems dedicated to scene understanding.