FusionBERT: Multi-View Image-3D Retrieval via Cross-Attention Visual Fusion and Normal-Aware 3D Encoder

Abstract: We propose FusionBERT, a novel multi-view visual fusion framework for image-3D multimodal retrieval. Existing image-3D representation learning methods predominantly focus on feature alignment of a single object image and its 3D model, limiting their applicability in realistic scenarios where an object is typically observed and captured from multiple viewpoints. Although multi-view observations naturally provide complementary geometric and appearance cues, existing multimodal large models rarely explore how to effectively fuse such multi-view visual information for better cross-modal retrieval. To address this limitation, we introduce a multi-view image-3D retrieval framework named FusionBERT, which innovatively utilizes a cross-attention-based multi-view visual aggregator to adaptively integrate features from multi-view images of an object. The proposed multi-view visual encoder fuses inter-view complementary relationships and selectively emphasizes informative visual cues across multiple views to get a more robustly fused visual feature for better 3D model matching. Furthermore, FusionBERT proposes a normal-aware 3D model encoder that can further enhance the 3D geometric feature of an object model by jointly encoding point normals and 3D positions, enabling a more robust representation learning for textureless or color-degraded 3D models. Extensive image-3D retrieval experiments demonstrate that FusionBERT achieves significantly higher retrieval accuracy than SOTA multimodal large models under both single-view and multi-view settings, establishing a strong baseline for multi-view multimodal retrieval.

• The paper introduces a novel two-branch framework combining a cross-attention multi-view image aggregator with a normal-aware 3D encoder to enhance retrieval accuracy.
• The paper demonstrates significant improvements, achieving a Top-1 recall of 68.73% on Objaverse-LVIS and outperforming existing single-view methods.
• The paper’s approach effectively handles textureless and no-RGB models by integrating surface normals, ensuring robust and reliable 3D matching.

FusionBERT: Multi-View Image–3D Retrieval via Cross-Attention Visual Fusion and Normal-Aware 3D Encoder

Problem Setting and Motivation

The task of matching 2D images to corresponding 3D models is central to numerous computer vision applications, including robotics, digital asset management, and AR content retrieval. Existing state-of-the-art approaches primarily employ single-image contrastive learning for image–3D representation alignment, typically neglecting the wealth of structural and appearance cues available when multiple images are captured from diverse views. Moreover, many current 3D model encoders critically depend on color features, posing challenges in contexts involving textureless CAD models or raw LiDAR scans where RGB channels are absent or degraded.

FusionBERT proposes a comprehensive solution specifically targeting the limitations of prior art in multi-view fusion and 3D geometry representation. By designing both a robust cross-attention-based aggregator for multi-view images and a surface normal-enriched 3D encoder, FusionBERT aims to advance retrieval accuracy and generalize more robustly across variable object appearances and modalities.

Architecture Overview

FusionBERT adopts a two-branch framework:

• Multi-View Visual Aggregator: A cross-attention mechanism fuses features from multiple image views, emphasizing inter-view complementarity and adaptively weighting images based on global and view-specific consistency.
• Normal-Aware 3D Encoder: Built atop Point-BERT, this encoder augments the standard positional and color input with surface normals, improving the fidelity and semantic richness of the learned 3D representations, especially when color information is missing or unreliable.

Both modules are trained to produce aligned embeddings in a contrastive tri-modal (image, text, 3D) space, following a two-stage alignment protocol.

Technical Contributions

Cross-Attention Multi-View Fusion: Image features from a pre-trained CLIP-ViT backbone are first refined using a self-attention Transformer encoder to model inter-view dependencies. Subsequently, a consensus embedding is computed as the mean feature, then used as a query in a cross-attention fusion module that outputs the final visual descriptor. This two-level attention structure systematically outperforms naive pooling methods by both leveraging view redundancy and suppressing outliers due to occlusion or clutter.

Normal-Enriched Point Cloud Encoder: By incorporating $[x, y, z, R, G, B, N_x, N_y, N_z]$ per point, where $N_x,N_y,N_z$ are local surface normals, the encoder is able to distinguish between geometric structures with higher fidelity and reduce its dependence on texture channels. The encoder retains computational efficiency comparable to vanilla Point-BERT.

Two-Stage Cross-Modal Alignment: The first stage aligns the point cloud encoder with the CLIP image encoder using an InfoNCE objective; the second stage freezes the 3D encoder and aligns the multi-view aggregator's output to the 3D encoder's space.

Experimental Results

FusionBERT demonstrates marked improvements over TAMM, OpenShape, and ULIP-2 under both single-view and multi-view settings. On Objaverse-LVIS, FusionBERT achieves Top-1 recall of 68.73% in the 3-view case, a significant boost compared to 60.90% for TAMM and 56.02% for OpenShape. The performance gap widens further in scenarios where color information is absent (LVIS no-RGB), indicating the efficacy of the normal-aware 3D encoding in handling textureless assets.

Ablation results show that both the cross-attention multi-view aggregator and the normal-aware encoder independently yield substantial performance gains, with combined usage producing the highest retrieval rates.

Additionally, the method exhibits consistent improvement as more views are added for querying, surpassing all baselines for input view counts up to 12. FusionBERT maintains competitive inference times relative to prior models, with multi-view visual encoding and 3D encoding both remaining practical for large-scale retrieval scenarios.

Implications and Future Directions

FusionBERT addresses fundamental gaps in multimodal retrieval pipelines by robustly aggregating multi-view 2D visual cues and leveraging surface geometry information insensitive to texture variance. This enables more accurate and reliable image-to-3D matching in real-world scenarios involving occlusion, clutter, and incomplete data. As object observations in practical deployments routinely cover multiple views, the approach offers a scalable and structurally sound foundation for real-world 3D asset retrieval.

Theoretically, FusionBERT highlights the importance of view-consistent feature learning and modality-agnostic geometric encoding. The two-stage contrastive alignment scheme further illustrates a pathway for reducing domain shifts between 2D image and 3D geometry spaces.

Potential future developments include integrating adaptive view selection—dynamically selecting the most informative image subset for fusion—and extending the general framework to generative 3D tasks, such as mesh or point cloud synthesis conditioned on multi-view imagery. Scaling pretraining corpora to broader multimodal datasets could further improve generalization in open-world scenarios.

Conclusion

FusionBERT delivers a technically substantiated advancement in multi-view image–3D retrieval by jointly optimizing cross-attention visual fusion and normal-aware geometric encoding. Extensive empirical evaluation validates its superiority over prevailing baselines, particularly in challenging, practical retrieval tasks where conventional models falter. The method sets a strong multimodal baseline and motivates further research on structured fusion and geometric reasoning across visual and spatial modalities (2604.02583).