S-MUSt3R: Sliding Multi-view 3D Reconstruction

Abstract: The recent paradigm shift in 3D vision led to the rise of foundation models with remarkable capabilities in 3D perception from uncalibrated images. However, extending these models to large-scale RGB stream 3D reconstruction remains challenging due to memory limitations. This work proposes S-MUSt3R, a simple and efficient pipeline that extends the limits of foundation models for monocular 3D reconstruction. Our approach addresses the scalability bottleneck of foundation models through a simple strategy of sequence segmentation followed by segment alignment and lightweight loop closure optimization. Without model retraining, we benefit from remarkable 3D reconstruction capacities of MUSt3R model and achieve trajectory and reconstruction performance comparable to traditional methods with more complex architecture. We evaluate S-MUSt3R on TUM, 7-Scenes and proprietary robot navigation datasets and show that S-MUSt3R runs successfully on long RGB sequences and produces accurate and consistent 3D reconstruction. Our results highlight the potential of leveraging the MUSt3R model for scalable monocular 3D scene in real-world settings, with an important advantage of making predictions directly in the metric space.

• The paper presents a sliding-window partition-align-optimize framework that enables scalable, uncalibrated 3D reconstruction using transformer-based models.
• The approach employs confidence-weighted local alignment and Lie group-based transform alignment with lightweight loop closure to reduce drift and ensure global consistency.
• Experimental results show a state-of-the-art average pose error of 0.052 m on the TUM dataset, proving its practical robustness in challenging robotic navigation scenarios.

Sliding Multi-view 3D Reconstruction with S-MUSt3R

Introduction

The S-MUSt3R framework addresses the persistent challenge of scalable, uncalibrated monocular 3D reconstruction from long RGB image sequences, a fundamental requirement in robotics and embodied vision. Conventional geometric or learning-based SLAM and SfM systems typically require either camera calibration, multiple sensing modalities, or complex, memory-intensive pipelines which limit practicality for long unsegmented sequences. Emergent transformer-based foundation models such as MUSt3R and VGGT have yielded superior local reconstructions, but their scalability is fundamentally limited by GPU memory constraints inherent to self-attention and memory mechanisms. S-MUSt3R extends the utility of foundation models to kilometer-scale sequences through a partition-align-optimize paradigm that maintains accuracy, efficiency, and practical deployment in real robotic settings (2602.04517).

S-MUSt3R Pipeline: Architecture and Algorithmic Contributions

At its core, S-MUSt3R builds upon the MUSt3R model, leveraging its capacity for accurate metric scene and pose predictions from uncalibrated RGB images. The novel pipeline segments long sequences into overlapping blocks, reconstructs each independently using MUSt3R, and then aligns and merges segment reconstructions through confidence-weighted pointmap and pose correspondences. Accumulated drift is corrected via lightweight loop closure, constructing a pose graph where segments are nodes and edges represent sequential or loop-derived alignment constraints.

Figure 1: Example of a pose graph where segments are nodes (blue) and loop closures (green), with edges encoding alignment constraints.

Segment Partitioning and Overlapping

The input RGB sequence is divided into overlapping segments of fixed length to localize MUSt3R memory requirements and maximize contextual correspondence between adjacent blocks. Overlap is critical both for redundancy in geometric cues and for robust filtering of local depth/prediction inconsistencies.

Confidence-weighted Alignment

MUSt3R provides depth and per-pixel confidence maps for each processed frame. S-MUSt3R innovates by modulating these confidences using depth agreement within overlapping regions, down-weighting correspondences for outlier patches, which substantially reduces alignment error in the segment stitching phase.

Lie Group-based Transform Alignment

Segment alignment is formulated on Lie groups of increasing expressiveness: SIM(3), Affine(3), and SL(4). While Affine(3) and SL(4) allow for non-uniform scaling and homography-based corrections (handling uncalibrated and projective distortions, respectively), empirical results demonstrate SIM(3) offers an optimal speed-accuracy-complexity trade-off in this architecture.

Loop Closure and Global Optimization

To eliminate accumulated drift, the pipeline efficiently detects loop closures via global image descriptor similarity (derived from the MUSt3R encoder), generating loop segments when scene content is revisited. Pose graph optimization is conducted on a segment-level graph (substantially reducing the total variable count relative to frame-level optimization), using Levenberg–Marquardt to jointly satisfy alignment constraints from sequential and loop closure matches.

Figure 2: 3D reconstruction with S-MUSt3R for TUM and 7-Scenes datasets. Top: RGB point cloud; Bottom: colored segment maps highlighting segment alignment.

Experimental Evaluation

Quantitative Results

Evaluation on TUM RGB-D and 7-Scenes datasets demonstrates that S-MUSt3R achieves competitive RMSE of absolute pose error (APE) and angular error (AAE) compared to pipelines with significantly more complex or memory-hungry graphical optimization backends, such as MASt3R-SLAM and VGGT-SLAM, and outperforms stitching-based methods like VGGT-Long by a substantial margin.

Notably, S-MUSt3R achieves the lowest average pose error (0.052 m) on the TUM dataset among all uncalibrated methods, despite the simplicity of its optimization approach, and is on par with or superior to graph-backend approaches on 7-Scenes.

Robustness in Real-world Navigation

On a proprietary robot navigation dataset—characterized by featureless corridors and challenging visual ambiguity—S-MUSt3R maintains trajectory accuracy where other methods collapse, due to the redundancy and corrective power of segment overlaps and loop closure.

Figure 3: Segment pose correction via lightweight loop closure on TUM and 7-Scenes datasets, demonstrating drift elimination and global consistency post-optimization.

Ablation Studies

Comprehensive ablations reveal several insights:

• Longer segment lengths reduce reconstruction error, as a higher context window limits edge effects, up to MUSt3R's feasible memory ceiling.
• SIM(3) provides globally optimal alignment results for segment stitching in the pipeline, with higher complexity groups offering minimal or inconsistent gains.
• Global metric scale is directly predicted, avoiding the need for extrinsic scale correction, an advantage over methods such as VGGT-Long.

Figure 4: 3D reconstruction from a robot navigation trajectory. Left/middle: S-MUSt3R predictions in RGB and segment color; right: VGGT-Long, which exhibits degraded structural accuracy.

Qualitative Analysis

Qualitative 3D reconstructions from both standard datasets and real navigation streams illustrate consistent and geometrically accurate scene predictions with S-MUSt3R. Visualizations with segment coloring confirm tight alignment and negligible discontinuities at segment boundaries. Loop closure corrections visibly eliminate drift, ensuring global geometric consistency even after long traversals.

Practical and Theoretical Implications

From a practical perspective, S-MUSt3R is directly deployable in robotic navigation and mapping scenarios requiring global metric estimates from monocular RGB alone, without preprocessing, extrinsic calibration, or significant computation overhead. The method's low memory footprint, enabled by the sliding-window strategy and segment-level optimization, offers scalability to kilometer-scale trajectories on commodity hardware.

Theoretically, the results demonstrate that modern transformer-based foundation models, when complemented by lightweight, confidence-driven graph alignment, can match or surpass the performance of classic multi-stage SLAM pipelines on challenging real-world data.

The ability to estimate metric scale directly from foundation models [cabon25muster] is highly advantageous for downstream tasks, eliminating the scale ambiguity and calibration barriers that previously limited monocular approaches.

Limitations and Future Directions

The stitching approach in S-MUSt3R is fundamentally limited by the quality of underlying local MUSt3R reconstructions. Segment-level alignment can only partially compensate for systematic depth or pose errors at the model level. Additionally, loop closure sensitivity to hyperparameters (e.g., similarity thresholds) may impact robustness, and could benefit from either self-adaptive mechanisms or probabilistic outlier modeling.

Potential future developments include:

• Integration with more expressive or multimodal foundation models (e.g., those incorporating inertial or depth sensors).
• Self-tuning hyperparameter modules for loop closure robustness.
• End-to-end differentiable pipelines blending trainable segmentation, alignment, and optimization.

Conclusion

S-MUSt3R demonstrates that robust, scalable, and metric 3D reconstruction from uncalibrated monocular RGB is achievable using transformer-based foundation models paired with a partition-align-optimize paradigm. The system delivers state-of-the-art accuracy on established benchmarks and demonstrates operational viability on real navigation data, all without complex optimization backends or extrinsic sensor data. This work substantiates the foundation model approach for embodied AI and opens further avenues for lightweight, globally consistent 3D vision pipelines in real-world robotics (2602.04517).