Efficient RGB-D Encoders

• Efficient RGB-D encoders are architectural paradigms that fuse complementary RGB and depth features while minimizing computational cost for real-time applications.
• They incorporate dual-branch CNNs, heterogeneous Transformers, and specialized fusion modules like attention and quality-aware gating to balance semantic and geometric representation.
• Key efficiency strategies include backbone compression, windowed self-attention, and hardware-focused optimizations, enabling robust performance in robotics and embodied perception.

Efficient RGB-D encoders are architectural paradigms and algorithmic systems designed to extract and fuse complementary features from RGB (color) and D (depth) modalities while minimizing computational and memory overhead. Efficiency is paramount for deployment in robotics, real-time scene analysis, and embedded applications, where constraints on latency, parameters, and FLOPs must trade against maximizing semantic, geometric, and cross-modal representational power.

1. Principles of Architecture Design for Efficient RGB-D Encoders

Efficient RGB-D encoders span dual-branch CNNs, heterogeneous Transformer backbones, and hybrid lightweight fusion models. In conventional dual-stream CNNs, both RGB and depth input are processed by separate backbones (often shallow ResNet-style for efficiency), and their features are fused at multiple stages with attention or additive operations (Seichter et al., 2020, Zhang et al., 2023). Transformative approaches such as single-stream RGB-D Transformers use unified patch-embedding convolution to jointly ingest RGB and depth as a 4-channel tensor—commonly in Swin-V2 configurations (Fischedick et al., 2023, Fischedick et al., 1 Jan 2026).

Specialized architectures further minimize redundancy: HDBFormer employs dual RGB encoders—a CNN for local detail, a Swin Transformer for global context—paired with a lightweight LDFormer for depth, using only depthwise separable convolutions and aggressive channel reduction. In DFM-Net, the Tailored Depth Backbone (TDB) is a five-stage MobileNet-V2 IRB subnetwork with ~0.9 MB size, compared to typical 7–10× larger backbones for RGB (Zhang et al., 2021, Zhang et al., 2022).

Key architectural strategies include:

• Multi-Scale Parallelism: Both RGB and depth generated at matched spatial resolutions—for fusion at each hierarchy (Seichter et al., 2020, Zhang et al., 2023, Zhang et al., 2021).
• Depth-Branch Minimalism: Dedicated, much shallower and narrower networks for depth reduce model size and computational cost by >90% compared to RGB backbones (Wei et al., 18 Apr 2025).
• Single-Stream Patch Embedding: Transformer-based models project RGB+D together into token space via a 4×4 convolutional patch-embedding, bypassing dual-branch redundancy (Fischedick et al., 1 Jan 2026).
• Asymmetric Capacity Allocation: Allocate greater model complexity (e.g., Transformers) for visually rich RGB, leveraging lightweight CNNs for geometrically focused depth (Wei et al., 18 Apr 2025).

2. Cross-Modal Fusion Mechanisms and Attention

Efficient RGB-D encoders exploit cross-modal fusion at multiple encoder stages, employing additive, attentive, or confidence-based mechanisms, with specific emphasis on data and modality-aware gating.

Common cross-modal fusion mechanisms:

• Additive Fusion with Attention: Outputs from both modalities are reweighted via Squeeze-and-Excitation (SE) blocks and simply summed at each stage (Seichter et al., 2020).
• Attention Fusion Modules (AFM): Channel and spatial attention blocks highlight informative channels and spatial regions for both RGB and depth, before summation (Zhang et al., 2023).
• Quality-Aware Fusion (DQFM): Depth features are gated by both a global quality scalar (α) and a spatial attention mask (β), dynamically suppressing low-quality/degraded depth input (Zhang et al., 2021, Zhang et al., 2022). DQFM relies on low-level alignment between early RGB and depth features, estimated via boundary alignment statistics interposed through a lightweight MLP. DHA computes β from deep depth features, fused with edge cues from RGB-depth.
• Modality Information Interaction Module (MIIM): Applies global fusion attention (scaled dot-product) and local fusion with large-kernel convolutions to combine RGB and depth at both long and short spatial ranges. This module is crucial for heterogeneous encoders with dramatically different RGB and depth processing regimes (Wei et al., 18 Apr 2025).
• Transformer Cross-Branch Fusion: In Vanishing Depth, fusion occurs token-wise between a frozen RGB ViT and a trainable depth ViT using SE modules at intermediate layers, preserving pre-trained semantic representations while enabling depth adaptation (Koch et al., 25 Mar 2025).

Fusion is categorized into early, progressive, and late variants. Early fusion forcibly aligns RGB/D within the dimensionally expanded input or at a shallow patch embedding layer, while progressive approaches integrate features at multiple resolutions/hierarchies. Late fusion (e.g., separate ViT forward passes with subsequent pooling/concatenation of the [CLS] token) maintains maximal independence between modalities up to the decision stage and has demonstrated superior sample efficiency and robustness in low-data scenarios (Tziafas et al., 2022).

3. Efficiency Strategies: Parameter, Latency, and FLOP Reduction

Efficiency is achieved through a combination of architectural compression, hardware-friendly operations, and dynamic gating. Notable methods include:

• Backbone Compression: LDFormer (depth) utilizes only depthwise-separable and pointwise convolutions, yielding 0.7 M parameters and 2.9 GFLOPs, compared to >25 M and >25 GFLOPs for ResNet-50 (Wei et al., 18 Apr 2025).
• Factorized Convolutions: Non-Bottleneck-1D replaces full 3×3 kernels with 3×1 and 1×3, cutting FLOPs by 33% per block (Seichter et al., 2020, Zhang et al., 2023).
• Windowed Self-Attention: Swin Transformers restrict attention to local windows—O(N·C·W²) vs. O(N²)—resulting in high real-time throughput on mobile GPUs (e.g., 39.1 FPS @ 640×480 on Jetson AGX Orin) (Fischedick et al., 2023, Fischedick et al., 1 Jan 2026).
• TensorRT Optimization: Exported ONNX models comprising only supported primitives (Conv, BN, ReLU, pooling, element-wise) can be deeply fused and parallelized for 4–5× latency reduction compared to unoptimized PyTorch (Seichter et al., 2020, Fischedick et al., 2023).
• Two-Stage Decoders: Stage-wise reduction in channel and hierarchy (as in DFM-Net) aggressively compresses required memory before final upsampling and prediction (Zhang et al., 2021).
• Late Fusion in Transformers: Zero parameter increase for the ViT/Transformer blocks; only fusion head incurs additional cost. This contrasts with early fusion, which (at least in dual-patch embedder variants) introduces another linear projection (Tziafas et al., 2022).
• Distillation and Knowledge Transfer: eVGGT distills a 24-layer ViT-L geometry-aware teacher to a 4-layer ViT-S student, achieving 9× speedup and 5× compression with minimal loss in absolute pose and depth reconstruction metrics (Vuong et al., 19 Sep 2025).

The following table summarizes exemplar model characteristics (as reported in the literature):

Model	Param Count	FLOPs (input size)	FPS	mIoU NYUv2	Distinguishing Features
ESANet-R34	~44 M	~70 G (640×480)	29.7	50.3%	Dual ResNet, SE fusion, TensorRT
DFM-Net (lite)	2.1 M	2.4 G (256×256)	64*	—	Tailored depth, DQFM, 2-stage dec.
EMSAFormer	30 M	12.2 G (640×480)	39.1	51.3%	Single SwinV2, window MSA, TRTOp
HDBFormer(LD)	0.7 M (D)	2.9 G (224×224)	—	59.3%†	LDFormer, MIIM, dual RGB encoders
DVEFormer	33 M	55 G (640×480)	26.3	57.1%	SwinV2-T, CLIP distillation
VanishingDepth	175 M*	131 G (224×224)	28*	56.1%	Dual frozen ViT, PDE, SE fusion
eVGGT	260 M	—	14*	—	ViT-S, geometry distillation

** FPS or param count context-dependent (varies by hardware and batch). †mIoU for encoder/decoder, not single stream.

4. Performance and Trade-offs in Benchmarks

Performance comparisons demonstrate that carefully designed RGB-D encoders can achieve state-of-the-art accuracy at significantly reduced parameter and latency budgets.

• On NYUv2 and SUN-RGBD, HDBFormer + LDFormer achieves 59.3% mIoU with 0.7 M param/2.9 GFLOP depth stream, outmatching ResNet-50 depth by >90% efficiency (Wei et al., 18 Apr 2025).
• DFM-Net variants attain top-2 accuracy among both efficient and large RGB-D SOD models on six datasets, while running at 20–60 FPS on CPU with <9 MB size (Zhang et al., 2021, Zhang et al., 2022). Ablation studies confirm the necessity of DQFM's scalar and spatial depth gating to maintain accuracy with tiny depth networks.
• Transformer-based EMSAFormer matches or slightly surpasses multi-task SOTA on NYUv2, ScanNet, and SUNRGB-D, while reaching 39.1 FPS at moderate parameter cost (Fischedick et al., 2023).
• In 3D object recognition, ViT-based late fusion methods establish new SOTA on ROD (95.4% top-1, Swin-L, late fusion, surface normal depth) and are empirically superior to early fusion by 5–8% (Tziafas et al., 2022).
• Vanishing Depth enables non-finetuned, frozen-encoder SOTA for segmentation (SUN-RGBD: 56.05 mIoU), depth completion (VOID: 88.3 mm RMSE), and 6D pose estimation (YCB-Video: 81.8% AUC ADD-S), via self-supervised depth encoding without added RGB encoder cost (Koch et al., 25 Mar 2025).
• HiPose leverages a hierarchical binary surface encoding with iterative correspondence pruning for 6D pose, achieving 84.6% mean BOP at 0.16 s/image, outperforming all refinement-free RGB-D pipelines at real-time speeds (Lin et al., 2023).
• eVGGT, distilled from VGGT, raises robot manipulation success up to 6.5% over standard vision encoders, with ~9× latency reduction (Vuong et al., 19 Sep 2025).

5. Applications and Use Cases in Robotics and Embodied Perception

Efficient RGB-D encoders underpin numerous robotic and embodied AI applications where both geometric and semantic scene understanding are essential and inference demands real-time or low-power execution.

• Interactive Robot Learning: ViT-based late fusion encoders integrate directly into lifelong object learning pipelines, allowing a robot to learn and recall novel object categories from as little as one demonstration, with 0% error over multiple real-world trials (Tziafas et al., 2022).
• Multi-task Scene Analysis: EMSAFormer unifies semantic/instance segmentation, object orientation, and scene classification in a single RGB-D backbone, suitable for mobile robot operation (Fischedick et al., 2023).
• Dense Visual Embeddings for 3D Mapping: DVEFormer produces text-aligned per-pixel embeddings enabling both classical segmentation and open-set text-based querying in dynamic 3D maps (Fischedick et al., 1 Jan 2026).
• Salient Object Detection: DFM-Net's low-latency gating technology enables mobile and real-time salient region detection, supporting assistive vision and AR systems (Zhang et al., 2021, Zhang et al., 2022).
• 6D Object Pose Estimation: HiPose's code-based RGB-D fusion yields highly accurate and robust 3D correspondences for manipulation, inventory, and tracking without the need for iterative refinement (Lin et al., 2023).
• Imitation Learning and Policy Conditioning: eVGGT embeddings improve the geometrical consistency of manipulation policies without explicit 3D supervision, increasing both simulation and real-world task success (Vuong et al., 19 Sep 2025).

6. Training Paradigms and Optimization

Optimizing for efficiency in RGB-D encoding requires not only smaller and shallower architectures but also data curricula, augmentation protocols, and fusion-aware regularization.

• Self-Supervised and Distillation Recipes: Models such as Vanishing Depth and DVEFormer exploit scale-invariant, multi-scale reconstruction and cosine distillation losses (from teacher models like CLIP or VGGT) to shape depth-aware representation spaces without high annotation cost or pretraining pipeline complexity (Koch et al., 25 Mar 2025, Fischedick et al., 1 Jan 2026, Vuong et al., 19 Sep 2025).
• Augmentation and Domain Invariance: Random depth scaling, masking, and global density shifts ensure robustness to diverse sensors and distributional drift (Koch et al., 25 Mar 2025). Progressive Perlin-based density masking explicitly regularizes models for sparse or corrupted depth.
• Late Fusion for Sample Efficiency: By decoupling depth processing from the frozen RGB weights, late fusion enables rapid adaptation to new (small) RGB-D datasets while avoiding overfitting (Tziafas et al., 2022).
• Parameter Regularization: For resource-constrained setups, depth branch channel counts and expansion ratios are systematically reduced, and computational bottlenecks are eliminated via factorized/depthwise operators (Zhang et al., 2021, Seichter et al., 2020, Wei et al., 18 Apr 2025).

7. Guidelines, Limitations, and Future Directions

Current literature converges on several guidelines for designing efficient RGB-D encoders:

1. Exploit modality asymmetry: retain expressive architectures for RGB, deploy minimal depth networks (Wei et al., 18 Apr 2025, Zhang et al., 2021).
2. Use progressive gating and spatial attention to suppress unreliable depth, especially in low SNR or sparsity regimes (Zhang et al., 2021, Zhang et al., 2022).
3. Fuse modalities at multiple encoder scales; avoid single-point "late" fusion except where warranted by sample constraints (Zhang et al., 2023).
4. Use windowed attention and hardware-aware deployment strategies (TensorRT, ONNX), especially for mobile robotics (Fischedick et al., 2023, Seichter et al., 2020).
5. Preserve backbone generality: adapters and fusion should not overwrite pretrained weights unless necessitated by strong cross-modal symmetry (Koch et al., 25 Mar 2025).

Limitations include persistent ambiguity in broad/unstructured semantic categories, memory overhead for high-dimensional embeddings (as in DVEFormer), and modest but irreducible parameter/FLOP cost for true dual-branch architectures. Promising directions include adapter-based modularity for arbitrary RGB backbones, joint learning of spatial, semantic, and depth cues with minimal supervision, and domain-invariant encoding across diverse sensors and scenes.

---

• "Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognition" (Tziafas et al., 2022)
• "Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis" (Seichter et al., 2020)
• "Depth Quality-Inspired Feature Manipulation for Efficient RGB-D Salient Object Detection" (Zhang et al., 2021)
• "Efficient Prediction of Dense Visual Embeddings via Distillation and RGB-D Transformers" (Fischedick et al., 1 Jan 2026)
• "Vanishing Depth: A Depth Adapter with Positional Depth Encoding for Generalized Image Encoders" (Koch et al., 25 Mar 2025)
• "HiPose: Hierarchical Binary Surface Encoding and Correspondence Pruning for RGB-D 6DoF Object Pose Estimation" (Lin et al., 2023)
• "Efficient Multi-Task Scene Analysis with RGB-D Transformers" (Fischedick et al., 2023)
• "Depth Quality-Inspired Feature Manipulation for Efficient RGB-D and Video Salient Object Detection" (Zhang et al., 2022)
• "Spatial-information Guided Adaptive Context-aware Network for Efficient RGB-D Semantic Segmentation" (Zhang et al., 2023)
• "Improving Robotic Manipulation with Efficient Geometry-Aware Vision Encoder" (Vuong et al., 19 Sep 2025)
• "HDBFormer: Efficient RGB-D Semantic Segmentation with A Heterogeneous Dual-Branch Framework" (Wei et al., 18 Apr 2025)