Explicit Memory through Online 3D Gaussian Splatting Improves Class-Agnostic Video Segmentation

Abstract: Remembering where object segments were predicted in the past is useful for improving the accuracy and consistency of class-agnostic video segmentation algorithms. Existing video segmentation algorithms typically use either no object-level memory (e.g. FastSAM) or they use implicit memories in the form of recurrent neural network features (e.g. SAM2). In this paper, we augment both types of segmentation models using an explicit 3D memory and show that the resulting models have more accurate and consistent predictions. For this, we develop an online 3D Gaussian Splatting (3DGS) technique to store predicted object-level segments generated throughout the duration of a video. Based on this 3DGS representation, a set of fusion techniques are developed, named FastSAM-Splat and SAM2-Splat, that use the explicit 3DGS memory to improve their respective foundation models' predictions. Ablation experiments are used to validate the proposed techniques' design and hyperparameter settings. Results from both real-world and simulated benchmarking experiments show that models which use explicit 3D memories result in more accurate and consistent predictions than those which use no memory or only implicit neural network memories. Project Page: https://topipari.com/projects/FastSAM-Splat/

• The paper demonstrates that explicit 3D Gaussian splatting memory, integrated with FastSAM and SAM2 models, improves accuracy and temporal consistency in video segmentation.
• The methodology incrementally updates a 3D memory using Gaussian features that encode spatial, color, and segment ID information, optimized via contrastive loss.
• Experimental results indicate significant gains in VSQ and STQ metrics, while the re-prompting mechanism in SAM2-Splat effectively corrects segmentation inconsistencies.

Explicit 3D Gaussian Splatting as Explicit Memory for Class-Agnostic Video Segmentation

Introduction

This work addresses the challenge of achieving temporally consistent, class-agnostic video segmentation in open-world robotic settings. The authors propose augmenting segmentation models with an explicit, online 3D memory using Gaussian Splatting (3DGS), enabling improved accuracy and consistency over models relying solely on implicit or no memory. The approach is instantiated in two models: FastSAM-Splat, which extends the image-based FastSAM, and SAM2-Splat, which augments the video-based SAM2 with explicit 3D memory and a re-prompting mechanism. The paper provides a comprehensive evaluation on real-world and simulated datasets, demonstrating that explicit 3DGS memory yields significant improvements in segmentation consistency and accuracy.

Methodology

Online 3D Gaussian Splatting as Explicit Memory

The core innovation is the use of an online 3DGS representation to store and update object-level segment predictions over time. Each Gaussian in the 3DGS encodes spatial position, orientation, scale, color, opacity, and a segment ID feature vector. The segment ID is drawn from a codebook optimized via a contrastive loss to maximize inter-segment separability. The 3DGS is constructed incrementally: at each frame, new Gaussians are initialized for pixels not covered by existing memory, and segment IDs are assigned based on the segmentation model's output.

This explicit memory can be rendered into both RGB and segment ID feature maps, which are then used to inform and refine future segmentation predictions.

FastSAM-Splat: Memory-Augmented Image Segmentation

FastSAM-Splat extends the FastSAM image segmentation model by integrating the 3DGS memory. At each frame, the 3DGS is rendered to produce 2D segment masks, which are matched to FastSAM's predictions using the Hungarian algorithm to maximize F-score. The fusion process handles matched, unmatched, and missing segments, updating the 3DGS memory accordingly. The segment ID features are updated via a loss combining magnitude MSE and cosine similarity, ensuring alignment between the memory and current predictions.

Figure 1: FastSAM-Splat model architecture, showing the integration of image-level segment predictions with the 3DGS memory.

SAM2-Splat: Memory-Augmented Video Segmentation with Re-Prompting

SAM2-Splat augments the SAM2 video segmentation model, which already uses implicit RNN-based memory, by incorporating explicit 3DGS memory. The key mechanism is a re-prompting strategy: at each frame, the 3DGS is rendered and compared to SAM2's predictions to identify three types of inconsistencies—missed tracks, incorrect track IDs, and duplicated tracks. For each inconsistency, SAM2 is re-prompted with positive/negative points sampled from the relevant regions, correcting errors and updating the memory.

Figure 2: SAM2-Splat model, illustrating the use of splat-segments from 3DGS memory and their integration with predicted image segments.

Figure 3: SAM2-Splat re-prompting strategy, showing how memory-based inconsistencies are corrected via targeted prompts.

Experimental Results

Real-World and Simulated Benchmarks

The models are evaluated on ScanNet-MV (real-world) and MVPd (simulated) datasets, using VSQ and STQ metrics to assess segmentation accuracy and temporal consistency. FastSAM-Splat achieves a +5.82% VSQ and +10.22% STQ improvement over FastSAM on ScanNet-MV. SAM2-Splat yields a +2.73% VSQ and +1.58% STQ improvement over SAM2 with YOLOv8 detections, and a +4.00% VSQ and +5.36% STQ improvement over SAM2 with ground truth detections. On MVPd, FastSAM-Splat attains the highest VSQ (56.76%), outperforming all baselines by at least +4.86%.

Figure 4: Qualitative comparison of FastSAM and FastSAM-Splat, highlighting reduced segment inconsistencies in FastSAM-Splat.

Figure 5: Comparison of SAM2 and SAM2-Splat, demonstrating reduced false negative flickering due to the re-prompting mechanism.

Ablation Studies

• Segment ID Feature Dimension: Increasing the segment ID vector dimension from 1 (integer) to 4 (real-valued) yields a substantial VSQ/STQ gain (+22.29%/+21.50%). Performance saturates beyond 14 dimensions, indicating diminishing returns for higher-dimensional embeddings.
• Re-Prompting Categories: Ablating the re-prompting strategy in SAM2-Splat shows that each category (missed, incorrect, duplicated tracks) contributes to improved consistency, with the combination of all three yielding the highest VSQ/STQ.
• Number of Re-Prompts: Using more than one re-prompt per object improves performance, but gains saturate beyond three prompts.

Efficiency Analysis

FastSAM-Splat and SAM2-Splat incur higher computational and memory costs compared to their baselines. FastSAM-Splat operates at 2.84 FPS, comparable to SAM2 (2.83 FPS), but slower than FastSAM (64.57 FPS). The explicit 3DGS memory requires 26–28M parameters per video, suggesting a trade-off between segmentation consistency and resource usage.

Discussion and Implications

The results demonstrate that explicit 3DGS memory significantly enhances both image-based and video-based segmentation models, with the largest gains observed for models lacking any temporal memory. The explicit memory mechanism is particularly effective in reducing temporal inconsistencies, such as false negative flickering and ID switches, which are critical for downstream robotic mapping and manipulation tasks.

The approach leverages the strengths of 3DGS for efficient, incremental scene representation, and the codebook-based segment ID embedding ensures robust segment association over time. The re-prompting strategy in SAM2-Splat provides a practical mechanism for correcting memory-induced errors in video segmentation.

However, the reliance on depth and pose inputs, as well as the increased memory and computational requirements, present challenges for deployment in resource-constrained settings. Future work should address memory optimization, global 3DGS refinement, and the extension to settings with partial or noisy depth/pose information.

Conclusion

This paper establishes that explicit, online 3D Gaussian Splatting memory provides substantial improvements in class-agnostic video segmentation consistency and accuracy. The FastSAM-Splat and SAM2-Splat models demonstrate that integrating explicit 3D memory with foundation segmentation models outperforms both image- and video-based baselines, particularly in open-world robotic scenarios. The findings suggest that explicit memory mechanisms will be increasingly important for robust, temporally consistent perception in embodied AI systems, and motivate further research into efficient, scalable memory representations and their integration with foundation models.