Dual-Input Convolutional Neural Network

• Dual-input CNNs are architectures that process two distinct data streams via parallel convolutional branches before fusing their features for joint inference.
• They employ strategies like parameter sharing, elementwise or attention-based fusion, and skip connections to boost accuracy and reduce overfitting.
• These networks are applied in computer vision, geometric deep learning, and signal processing, demonstrating improved performance in tasks such as segmentation and classification.

A dual-input convolutional neural network (CNN) refers to any deep neural architecture in which distinct convolutional branches, modules, or input streams process two separate (but potentially related) modalities, representations, or domains before fusing their representations for joint inference. Such networks are motivated by application settings where complementary information is available in structured form, including images plus gradients, audio plus spatial metadata, paired geometric descriptors (faces and edges), or visual–language pairs. The dual-input framework can exploit these complementary signals via architectural coupling, parameter sharing, and late or early fusion. Key empirical results in computer vision, geometric deep learning, and signal processing have demonstrated the utility of dual-input CNNs for improved accuracy and generalization across a range of tasks.

1. Architectural Principles of Dual-Input CNNs

Dual-input CNNs are defined by branches that operate on two parallel input streams. These may include two physical sensory modalities; original and derived feature representations (e.g., images, images gradients); or disparate data types, such as spatial maps and sequential/language descriptors.

Key architectural motifs include:

• Parallel branches: Separate convolutional (or hybrid) paths process each input independently, often with identical weights or mirrored block structures.
• Parameter sharing: In certain implementations, convolutional kernels and biases are shared across both streams to constrain the hypothesis space and promote domain-agnostic feature extraction (Pandey et al., 2020).
• Fusion layers: Intermediate or terminal node(s) combine the feature spaces. Fusion operations include elementwise addition, concatenation, or learnable attention-based mechanisms, depending on the semantic or statistical relationship between the two domains.
• Task-driven design: Decoder architectures, pooling, and downstream heads are adapted for task context—segmentation, classification, regression, or structured output.

Common use cases include vision with edge statistics (Pandey et al., 2020), vision–language grounding (Ye et al., 2020), geometric mesh tasks (Milano et al., 2020), and audio–metadata fusion (Grinstein et al., 2023).

2. Mathematical Formulations of Key Dual-Input Designs

Visual–Language Dual ConvLSTM

For referring image segmentation, a dual-input ConvLSTM architecture fuses a static spatial map and a word sequence (Ye et al., 2020). Let $I$ be an image; $w_1,\ldots,w_T$ a list of tokens in an expression. The process is:

• Extract $V\in\mathbb{R}^{C\times H\times W}$ via a CNN.
• Embed each $w_t$: $e_t\in\mathbb{R}^{D}$.
• At each time-step $t$:
  • Broadcast $e_t$ to $E_t\in\mathbb{R}^{D\times H\times W}$.
  • Form $x_t = [V; E_t]$ via channelwise concatenation.

The ConvLSTM update equations are:

$$\begin{aligned} i_t &= \sigma(W_{xi}*x_t + W_{hi}*h_{t-1} + W_{ci}\circ c_{t-1} + b_i) \ f_t &= \sigma(W_{xf}*x_t + W_{hf}*h_{t-1} + W_{cf}\circ c_{t-1} + b_f) \ c_t &= f_t \circ c_{t-1} + i_t\circ\tanh(W_{xc}*x_t + W_{hc}*h_{t-1} + b_c) \ o_t &= \sigma(W_{xo}*x_t + W_{ho}*h_{t-1} + W_{co}\circ c_t + b_o) \ h_t &= o_t\circ\tanh(c_t) \end{aligned}$$

with all $W_{*}$ learned convolutional kernels. This structure encodes sequential word–spatial fusion.

Image + Gradient (Edge) Dual-Input CNN

A two-stream CNN processes both raw image $I$ and a derived gradient map $G(x,y) = |K_x*I| + |K_y*I|$ (Sobel edges) (Pandey et al., 2020). The streams share all convolutional weights:

$$\begin{aligned} X_I^{(l)} &= \sigma(W^{(l)}*X_I^{(l-1)} + b^{(l)}) \ X_G^{(l)} &= \sigma(W^{(l)}*X_G^{(l-1)} + b^{(l)}) \ Z &= X_I^{(L)} + X_G^{(L)} \ \hat{y} &= \mathrm{Softmax}(W^{(d)}\mathrm{Flatten}(Z) + b^{(d)}) \end{aligned}$$

No increase in parameter count occurs due to weight sharing.

Dual Feature Mesh CNN

For 3D triangle meshes, the Primal–Dual MeshCNN (Milano et al., 2020) constructs two associated graphs: the primal (faces as nodes) and the dual (edges as nodes). Input features are:

• Primal: face-wise $\mathbf{f}_A$ (e.g., normalized area per face).
• Dual: edge-wise $\tilde{\mathbf{f}}_{\{A,B\}}$ (e.g., dihedral angle, normalized edge-lengths, corner angles).

At each layer, a dual convolution (graph attention) on edge features propagates to a primal convolution on faces, and vice versa, each modulated by the other's representation.

Audio + Metadata DI-NN

For planar sound source localization, one branch processes the high-dimensional, multi-channel audio $X\in\mathbb{R}^{2M\times L\times F}$ via 2D convolutions and recurrent units, producing $h_\mathrm{audio}\in\mathbb{R}^{256}$. A parallel branch passes metadata $\phi\in\mathbb{R}^{N_\phi}$ (microphone coordinates, room properties) through optional embedding FC layers to yield $h_\mathrm{meta}\in\mathbb{R}^{N_\phi}$. Feature vectors are concatenated and mapped by an MLP to the position prediction $\hat{p}\in\mathbb{R}^2$ (Grinstein et al., 2023).

3. Feature Fusion Strategies and Attention Mechanisms

Fusion in dual-input convolutional architectures ranges from simple elementwise operations to sophisticated attention-based interaction:

• Elementwise Addition/Concatenation: Feature tensors from both streams are added or concatenated. The operation is performed after an equal number of convolutional layers, immediately before task-specific dense heads (Pandey et al., 2020).
• Attention-based Fusion: For multimodal alignment (e.g., language–vision), attention mechanisms are employed to weight the relevance of each token or region. For instance, weighted pooling over embedded word tokens may be performed before integration with visual features. In geometric dual-input nets, the attention coefficients used in dual–primal graph convolutions modulate feature propagation and pooling (Milano et al., 2020).
• Skip Connections / Multi-level Feature Integration: In encoder–decoder settings, features from intermediate layers of one branch (e.g., visual backbone) may be “skipped” for multi-scale fusion, as in upsampling decoders. This corrects information loss from deep compressive layers and enables integration of fine-scale details (Ye et al., 2020).

4. Empirical Results and Benchmark Comparisons

Dual-input CNNs have demonstrated substantial performance improvements across a range of tasks:

Task / Dataset	Dual-Input Method	Baseline Acc. / Error	Dual-Input Acc. / Error	Gain	Source
MNIST (10-way)	Pixel + Gradient Shared CNN	98.2%	98.5%	+0.3	(Pandey et al., 2020)
CIFAR-10 (10-way)	Pixel + Gradient Shared CNN	88.7%	90.3%	+1.6	(Pandey et al., 2020)
CIFAR-100 (100-way)	Pixel + Gradient Shared CNN	56.1%	58.7%	+2.6	(Pandey et al., 2020)
SHREC 16/10 splits	Primal–Dual MeshCNN	98.6/91.0%	99.7/99.1%	+1.1/+8.1	(Milano et al., 2020)
COSEG Chairs	Primal–Dual MeshCNN	92.99%	97.23%	+4.24	(Milano et al., 2020)
Real SSL (audio)	CRNN (no metadata): 0.221 m	DI-NN: 0.121 m	Error halved	(Grinstein et al., 2023)

These results indicate that dual-input configurations yield consistent improvements over single-input baselines. The addition of complementary signal domains—when architecturally fused—reduces test error and sharpens task-specific outputs such as masks, class labels, or coordinate regression.

5. Task-Specific Design Considerations

Different application domains motivate distinct dual-input architectural variants:

• Referring Image Segmentation: Simultaneous spatial and linguistic input. A dual-input ConvLSTM encoder captures spatially resolved sequential language grounding (Ye et al., 2020). Multi-level decoder fusion enables precise boundary delineation.
• Image Classification with Edges: Raw pixels are augmented by edge/gradient maps in a shared-weights two-stream CNN, imparting stronger inductive bias with no increase in parameter count (Pandey et al., 2020).
• Geometric Deep Learning: PD-MeshNet alternates attention-driven convolutions on meshes’ faces and edges for accurate shape classification/segmentation, exploiting mesh topology in both primal and dual graph views (Milano et al., 2020).
• Signal Processing with Metadata: Planar sound source localization combines learned representations from both audio spectrograms and explicit scene metadata (microphone positions, room properties) in a composite network, offering explicit generalization to unseen configurations (Grinstein et al., 2023).

6. Generalization and Extensions of the Dual-Input Paradigm

The dual-input convolutional framework generalizes to any scenario involving complementary spatial, sequential, or structural modalities:

• Any spatial map (image, spectrogram, mesh, point cloud) may be paired with another representation (sequence, graph, auxiliary statistics).
• Non-visual branches might encode text, sensor metadata, graph embeddings, or derived features (HOG, Laplacian, style maps), fused via shared or coordinated feature extractors.
• The decoder or output head can be adapted for a variety of structured outputs: segmentation masks, heatmaps, detection boxes, or continuous coordinates.
• A plausible implication is that shared-parameter dual-input architectures act as regularizers, enforcing invariance to modality or representation-specific artifacts and reducing overfitting risk, provided the auxiliary stream is relevant.

7. Limitations, Insights, and Future Directions

Reported limitations include reliance on manually chosen feature transformations (such as $|dx|+|dy|$ for gradients) and simple fusion mechanisms (elementwise Add) (Pandey et al., 2020). Extension opportunities include:

• Incorporating richer or directional features (e.g., orientation histograms, learned descriptors) as additional streams.
• Applying attention-based or hierarchical fusion to better exploit cross-modal correlations in complex tasks.
• Extending the primal–dual paradigm to other irregular domains (e.g., volumetric data, point clouds with edge/face connectivity).
• Systematic evaluation of dual-input designs with varying degrees of parameter sharing to balance generalization and task-specific adaptation.

The dual-input convolutional architecture continues to serve as a unifying pattern for the integration of heterogeneous information within deep neural models, offering tangible improvements and new methodological avenues across domains as diverse as computer vision, geometric learning, and signal processing (Ye et al., 2020, Pandey et al., 2020, Milano et al., 2020, Grinstein et al., 2023).