Efficient RGB-D Encoder Overview

Updated 22 January 2026

Efficient RGB-D encoders are specialized modules that extract and fuse color and depth features using streamlined architectures to reduce computational complexity and latency.
They integrate diverse fusion strategies, including single-stream, dual-stream with cross-modal attention, and transformer-based designs to enhance performance in segmentation and robotics.
Empirical analyses indicate these encoders deliver competitive accuracy with significant reductions in parameters, FLOPs, and inference times on resource-constrained platforms.

Efficient RGB-D encoders are architectural modules designed to extract and fuse complementary features from color (RGB) and depth (D) modalities while minimizing computational complexity, parameter count, and inference latency. Recent advances combine architectural innovation, attention mechanisms, and modality-aware optimization to enable real-time deployment on resource-constrained platforms, with applications in semantic segmentation, salient object detection, scene understanding, and robotics.

1. Architectures and Fusion Strategies

Efficient RGB-D encoders follow three principal paradigms, each with distinct design trade-offs:

a) Single-Stream Early Fusion:

Depth is concatenated with RGB at the input stage, processed by a unified backbone, typically a CNN or Transformer. Kernels are specialized to handle the heterogeneity of modalities:

Example: In “A Single Stream Network for Robust and Real-time RGB-D Salient Object Detection,” a 4-channel input (RGB+D) is processed by a modified VGG-16, where the first convolution is extended to 4 channels (RGB weights from ImageNet, depth kernel He-initialized). Subsequent blocks share weights with standard VGG-16 (Zhao et al., 2020).
In efficient Swin-Transformer variants, early fusion is accomplished via a widened patch embedding layer; separate convolutions for RGB (C_r channels) and D (C_d channels) precede concatenation, preserving modality-specific representation in initial tokens (Fischedick et al., 2023, Fischedick et al., 1 Jan 2026).

b) Dual-Stream Encoders with Cross-modal Gating or Attention:

Parallel RGB and depth branches, each with tailored backbones, exchange features via attention-based fusion or learned gates after every hierarchical stage:

DFM-Net uses MobileNet-V2 for RGB and a lightweight, inverted residual bottleneck stack for depth; cross-modal fusion is modulated by Depth Quality-Inspired Feature Manipulation (DQFM), which controls the contribution of depth via learned global weights and spatial masks (Zhang et al., 2021, Zhang et al., 2022).
HDBFormer assigns a transformer backbone to RGB and a hierarchical depthwise-separable CNN ("LDFormer") to depth. Multi-level fusion occurs via modality-specific global/local attention modules (MIIM), which use both windowed self-attention and large-kernel convolution for robust interaction (Wei et al., 18 Apr 2025).

c) Transformer-Only or Adapter-Based Designs:

A single vision transformer (ViT) backbone handles both modalities or a depth adapter branches off from a frozen RGB transformer:

CoMAE employs a single ViT, with modality-specific linear patch embeddings, and fuses by stacking tokens; cross-modal interactions emerge naturally from self-attention (Yang et al., 2023).
Vanishing Depth attaches a trainable ViT depth adapter to a frozen RGB ViT (e.g., DINOv2), guided by a positional depth encoding (PDE) that ensures scale and density invariance. Gradual fusion occurs through Squeeze-and-Excitation gating at intermediate layers, with negligible cost (Koch et al., 25 Mar 2025).

Optimizing the utility of depth requires dynamically weighting its influence based on content and input quality:

Dual Attention Modules:

DEDA (Depth-Enhanced Dual Attention) integrates both mask-guided and depth-contrast attention to selectively enhance foreground/background discrimination in a single-stream network (Zhao et al., 2020).
DQFM in DFM-Net computes global hierarchy-wise weights (α_i) using low-level alignment (Dice-like metric between RGB, depth edge features) and generates spatial masks (β_i) through recursive fusion of high-level depth and low-level edge cues, suppressing low-quality depth while preserving salient structure (Zhang et al., 2021, Zhang et al., 2022).

Fusion Attention in Deeper/Transformer Designs:

MIIM in HDBFormer executes global attention using spatially pooled transformer-style queries, and local attention via 7×7 kernels plus gating, iteratively refining modality fusion (Wei et al., 18 Apr 2025).
Squeeze-and-Excitation modules are commonly used for channel-wise fusion, both in traditional CNN backbones and in ViT adapters (e.g., Vanishing Depth) (Seichter et al., 2020, Koch et al., 25 Mar 2025).

3. Hierarchical and Multi-Scale Context Modules

Effective semantic reasoning in RGB-D requires multiscale feature aggregation:

PAFE (Pyramidally Attended Feature Extraction): At the encoder’s top stage, parallel branches with increasing dilation rates and non-local attention capture multi-scale context, fusing output via 1×1 convolution to coalesce long-range and global cues (Zhao et al., 2020).
Adaptive Context (SGACNet and APC): Global context is modulated by affinity maps between local (dilated) and global branches, then upsampled for decoder input. Dual-branch attention modules combine channel (SE-style) and pyramid-pooling–based spatial masks at each hierarchical level (Zhang et al., 2023).

4. Parameter, FLOP, and Inference Analysis

Efficient encoder designs employ lightweight networks, depthwise-separable convolutions, and early fusion to minimize model size and compute:

Model	Parameter Count	Inference FPS (Resolution)	Notable FLOPs/Hardware
DFM-Net (Zhang et al., 2021)	2.53M (8.5 MB)	64 GPU / 20 CPU (256×256)	339M FLOPs
ESANet (Seichter et al., 2020)	~40M	29.7 (640×480, Jetson)	~15 GFLOPs (ResNet-34 dual-branch)
EMSAFormer (Fischedick et al., 2023)	28–32M	30.5–47.7 (640×480, Orin)	4.5G (Swin-Tiny, patch-embedded, FP16)
HDBFormer LDFormer	0.7M (depth)	—	2.9G (LDFormer), >20M for prior SOTA depth
DVEFormer (Fischedick et al., 1 Jan 2026)	~28M	26.3 (full), 77.0 (¼-res)	640×480 Orin, 768-D per-pixel outputs
Vanishing Depth (Koch et al., 25 Mar 2025)	+86M (PDE)	—	Adds ~2× base ViT-B/14 compute (frozen RGB)

Designs employing MobileNet-V2, custom lightweight depth backbones (TDB, LDFormer), or ViT-based adapters offer order-of-magnitude reductions in parameter count and FLOPs relative to prior dual-stream SOTA, while achieving comparable or higher accuracy.

Quantitative ablations consistently confirm that early or single-stream fusion, combined with quality-aware attention or adaptive fusion, yields substantial memory/compute savings and robust performance even on low-quality or missing depth data (Zhao et al., 2020, Zhang et al., 2021, Yang et al., 2023, Wei et al., 18 Apr 2025).

5. Empirical Performance and Application Domains

Efficient RGB-D encoders are validated across salient object detection, indoor semantic segmentation, open-vocabulary embedding, and dense 3D mapping.

Salient Object Detection: DFM-Net achieves state-of-the-art accuracy within 0.2% of much heavier models, at ~8× smaller size and 10× lower FLOPs; gating modules such as DQFM are critical in handling noisy or low-quality input (Zhang et al., 2021, Zhang et al., 2022, Zhao et al., 2020).
Semantic Segmentation: HDBFormer delivers 59.3% mIoU on NYUDepthv2 with a 0.7M parameter depth branch, outperforming dual-transformer baselines at <4% of the parameter/compute cost for depth (Wei et al., 18 Apr 2025). SGACNet attains 48.2–49.4% mIoU (RGB-D) with 30–50% fewer parameters and 15–20% lower latency than competitive ResNet-based dual-branch networks (Zhang et al., 2023).
Transformer Embedding Tasks: DVEFormer enables dense text-aligned pixel embeddings (CLIP-like) for both closed-set and open-vocabulary querying at >25 FPS and supports 3D semantic mapping pipelines (Fischedick et al., 1 Jan 2026).
Open Set/Robust Pretraining: CoMAE’s single-encoder strategy yields near-SOTA recognition with only 5K labeled images and is robust to missing modalities via modality-randomization (Yang et al., 2023). Vanishing Depth supports non-finetuned, generalizable RGB-D feature extraction through self-supervised training and sinusoidal positional depth encoding (Koch et al., 25 Mar 2025).

6. Ablation, Analysis, and Future Directions

Detailed empirical ablations across the literature establish:

Modality-aware attention and gating (DQFM, DEDA, MIIM) are essential for masking or robustly ignoring unreliable depth input, leading to substantial accuracy gains in challenging settings.
Lightweight depth backbones (LDFormer, TDB) suffice for depth cues; allocating heavy models to depth is inefficient.
Single-stream or transformer-unified early fusion (and transformer token mixing) outperforms late or naive mid-level fusion in both speed and generalization capacity.
Open-vocabulary and scene-adaptive extensions, particularly those leveraging dense text-image embedding distillation (CLIP, DVEFormer), are transforming the utility of efficient encoders by supporting new robotics and mapping use-cases (Fischedick et al., 1 Jan 2026, Koch et al., 25 Mar 2025).

Ongoing directions include more effective handling of sparse/noisy real-world depth (distribution/density-invariant encoding), broader deployment via modular acceleration (TensorRT, ONNX), and seamless integration with large pre-trained multimodal transformers.

7. Key References

W. Zhang et al., “A Single Stream Network for Robust and Real-time RGB-D Salient Object Detection” (Zhao et al., 2020)
S. Wu et al., “HDBFormer: Efficient RGB-D Semantic Segmentation with A Heterogeneous Dual-Branch Framework” (Wei et al., 18 Apr 2025)
J. Liu et al., “CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets” (Yang et al., 2023)
S. Sayed et al., “Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis” (Seichter et al., 2020)
W. Zhang et al., “Depth Quality-Inspired Feature Manipulation for Efficient RGB-D Salient Object Detection” (Zhang et al., 2021, Zhang et al., 2022)
J. Tagwerker et al., “Vanishing Depth: A Depth Adapter with Positional Depth Encoding for Generalized Image Encoders” (Koch et al., 25 Mar 2025)
H. Evermann et al., “Efficient Multi-Task Scene Analysis with RGB-D Transformers” (Fischedick et al., 2023)
M. Li et al., “Spatial-information Guided Adaptive Context-aware Network for Efficient RGB-D Semantic Segmentation” (Zhang et al., 2023)
T. Sayed et al., “Efficient Prediction of Dense Visual Embeddings via Distillation and RGB-D Transformers” (Fischedick et al., 1 Jan 2026)

Efficient RGB-D encoding designs are converging toward unified, lightweight architectures that maximize parameter sharing, leverage cross-modal attention, and exploit the latest advances in transformer-based modeling and multi-task learning for both embedded deployment and high-level general scene understanding.