DFS: Dual Feature Shift for Multi-Modal Fusion

• Dual Feature Shift (DFS) is a multi-stage, zero-cost neural mechanism that fuses complementary video modalities and temporal frames for improved driver action recognition.
• It alternates modality feature interaction and neighbour feature propagation shifts between CNN stages to enhance both intra- and inter-modal representations.
• DFS sets new benchmarks on the DriveAct dataset by delivering superior accuracy and lower latency with reduced computational overhead.

Dual Feature Shift (DFS) is a multi-stage, zero-cost neural mechanism for fusing complementary information from multiple synchronized video modalities and temporal frames, with the aim of improving accuracy and efficiency in driver action recognition (DAR) within vehicle cabin monitoring systems. DFS alternates cross-modality (modality feature interaction) and temporal (neighbour feature propagation) shift operations between convolutional neural network (CNN) backbone stages, enhancing both intra- and inter-modal representations while maintaining low computational overhead. It establishes new benchmarks for multi-modality DAR, particularly on the DriveAct dataset, where it significantly outperforms prior approaches in both accuracy and latency (Lin et al., 2024).

1. Background: Driver Action Recognition and Multi-Modality Fusion

Driver Action Recognition targets the automatic identification of fine-grained activities from vehicle cabin video streams, such as eating, drinking, or using a phone. The task is central to advanced driver-assistance systems and human–machine interaction safety. Single-modality approaches suffer from limited field‐of‐view, ambiguity between similar motions, and sensitivity to cabin lighting variability. Multi-modality distributed sensing—combining RGB, Infra-Red (IR), and Depth sequences—provides diverse information including texture, thermal, and 3D cues, thereby improving disambiguation and robustness.

2. DFS Mechanism: Mathematical Definition

Consider the multi-modal DAR input: $$D = \left\{ X^{(1)}, X^{(2)}, \dots, X^{(N)} \right\}, \quad X^{(m)} \in \mathbb{R}^{C \times T \times H \times W},$$ where $N$ is the number of synchronized modalities, $C$ the number of channels per modality, $T$ the number of temporal frames, and $(H, W)$ the spatial dimensions.

DFS applies two shift operations between CNN stages:

1. Modality Feature Interaction ($M_{\rm shift}$): For modalities $p$ and $q$ at time $t$, with per-frame features $$F^{(p)}_t, F^{(q)}_t \in \mathbb{R}^{C \times H \times W}$$ and swap width $k = C/8$,

$$\begin{aligned} \widehat{F}^{(p)}_t &= M_{\rm shift}(F^{(p)}_t, F^{(q)}_t) = \mathrm{Concat}(F^{(p)}_t[:, : C - k], F^{(q)}_t[:, C - k:]), \ \widehat{F}^{(q)}_t &= M_{\rm shift}(F^{(q)}_t, F^{(p)}_t) = \mathrm{Concat}(F^{(q)}_t[:, : C - k], F^{(p)}_t[:, C - k:]). \end{aligned} \tag{1}$$

2. Neighbour Feature Propagation ($T_{\rm shift}$): For modality $p$, frame $t$, channel width $i$ ($2i = C/4$),

$$\widetilde{F}^{(p)}_t = T_{\rm shift}(\widehat{F}^{(p)}_{t-1}, \widehat{F}^{(p)}_t, \widehat{F}^{(p)}_{t+1}) = \mathrm{Concat}\left( \widehat{F}^{(p)}_{t-1}[:, 0: i],\; \widehat{F}^{(p)}_{t+1}[:, i: 2i],\; \widehat{F}^{(p)}_{t}[:, 2i: C] \right). \tag{2}$$

The alternation of these operations propagates complementary modality cues and temporally local context through the network with negligible increase in memory access or computation.

3. Algorithmic Pipeline and Architecture

The DFS method employs a ResNet-50 backbone partitioned into five stages, with weight sharing in intermediate stages to foster modality-agnostic mid-level representation. The overall computation proceeds as follows:

• Stage Splitting: ResNet-50 is divided as conv1+conv2_x (Stage 1), shared conv3_x (Stage 2), shared conv4_x (Stage 3), modality-specific conv5_x (Stage 4), and final pooling/head (Stage 5).
• Input Processing: Each modality is processed as a tensor of shape $C \times T \times H \times W$, with temporal stride 8, temporal clip length $T=8$, and spatial crop/resize to $224 \times 224$.
• Dual Feature Shift: Between every two backbone stages, for each frame $t$ and modality $m$, features undergo $M_{\rm shift}$ with a partner modality (cyclic/pairwise), followed by $T_{\rm shift}$ along frames.
• CNN Encoding: Each stage processes features through its CNN block, with stages 2 and 3 using weight sharing across all modalities.
• Fusion and Output: Features from all modalities after Stage 5 are average-pooled (over modality and time), then passed to a fully connected classification layer for cross-entropy loss computation.

Key hyper-parameters include $k = C/8$ (modality shift channels), $2i = C/4$ (temporal shift channels).

4. Component Analysis: Cross-Modality and Temporal Shifts

3.1 Modality Feature Interaction

Early and multi-depth fusion is achieved by exchanging the last $k$ channels across feature maps from paired modalities, enabling the network to capitalize on complementary spaces (e.g., merging RGB texture with IR heat signatures). This "channel swap" incurs zero FLOPs and is performed before and after each weight-shared CNN stage, increasing the frequency and depth of cross-modal integration. Weight sharing in Stages 2 and 3 further encourages learning of modality-invariant representations.

3.2 Neighbour Feature Propagation

Temporal context is facilitated by shifting the first $i$ channels from the preceding frame and the next $i$ from the succeeding frame into the current feature map, with the remainder of the channels preserving the current frame's content. This redistribution enables bidirectional short-term motion cues to inform per-frame representations with zero multiplication-addition cost.

5. Empirical Evaluation and Comparative Results

DFS is evaluated on the DriveAct dataset using three predefined splits (right-top view, RGB+IR+Depth), averaging across results.

Comparative Accuracy (Depth+IR, Top-1 / Balanced)

Method	Top-1 (%)	Balanced (%)
ResNet-50 (late fusion)	56.43	51.08
TSM	70.31	61.11
MDBU (avg. fusion)	74.31	60.25
MDBU (max. fusion)	72.49	59.70
DFS	77.61	63.12

Modality Ablation

DFS demonstrates highest gains with IR+Depth (77.61 / 63.12), outperforming RGB+IR (72.32 / 62.87), RGB+Depth (73.15 / 62.67), and all single-modality settings.

Feature-Shift Ablation

Setting	Top-1 (%)	Balanced (%)
M+T, shared	77.61	63.12
T-only, shared	67.73	58.03
T-only, non-shared	70.31	61.11
No shift, no share	56.43	51.08

Both modality and temporal shifts, in conjunction with weight sharing, are essential for maximum performance.

Efficiency Metrics (Depth+IR)

Method	Latency (ms)	Params (M)
TSM	33	47.2
DFS	28	38.8

DFS achieves lower latency and reduced parameter count compared to TSM.

6. Training Protocols and Implementation Details

DFS initializes ResNet-50 weights from ImageNet pretraining. Training employs SGD with an initial learning rate of $10^{-4}$, momentum 0.9, and weight decay $5 \times 10^{-4}$. Cross-entropy loss is used with data augmentation incorporating random crop and horizontal flip. The architecture averages reported metrics over the three predefined splits of DriveAct.

7. Strengths, Constraints, and Prospective Extensions

DFS provides a modular, efficient mechanism for deep multi-modal and temporal integration with negligible computational cost. Multi-stage integration enables richer feature interaction than late or max pooling strategies. Weight sharing reduces parameter overhead and fosters learning of shared mid-level features across modalities.

Operational constraints include requirements for strict time-synchronization and spatial alignment of input modalities, as well as the use of a fixed channel swap budget ($k$), which may not fully capture dynamic cross-modality dependencies.

Potential extensions include shift-rate adaptation (making $k$ and $i$ learnable), replacing fixed channel slicing with attention-guided fusion, and application to other multi-sensor settings such as surveillance (RGB+thermal), robotics (RGB+depth+lidar), or medical video (RGB+thermal+flow).

DFS establishes a high-efficiency, high-accuracy baseline for multi-modality and temporal fusion in driver action recognition, with demonstrated superiority on DriveAct in both predictive performance and resource consumption (Lin et al., 2024).