Dynamic Nomination Decoders (DNDs) in DSCT

• Dynamic Nomination Decoders (DNDs) are advanced multi-modal routing modules that dynamically choose between region-level and segmentation-level features at each token.
• They integrate into DSCT by using separate cross-attention branches and a differentiable nomination mechanism (via Gumbel-softmax) to optimally fuse visual cues.
• Empirical evaluations demonstrate that DNDs yield higher metrics (e.g., CIDEr, B@4) compared to traditional fusion methods, addressing semantic and spatial misalignments.

Dynamic Nomination Decoders (DNDs) are advanced multi-modal routing modules introduced within the Dual-Stream Collaborative Transformer (DSCT) framework for image captioning. DNDs dynamically determine, at each token in the output caption and at each decoding layer, whether to utilize region-level or segmentation-level visual information, thereby bypassing the semantic inconsistencies and spatial misalignments frequently encountered when fusing disparate visual feature streams. This selection process is differentiable and operates at the granularity of both sequence position and network depth, resulting in measurably improved descriptive accuracy over prior fusion strategies (Wan et al., 19 Jan 2026).

1. Integration and Function of DND within DSCT

Within DSCT, visual features are processed along two distinct streams: region (object-centric) features and segmentation (context/global) features, first consolidated via Pattern-Specific Mutual Attention Encoders (PSMAEs). DNDs are positioned atop these encodings and the partial caption sequence. For each decoding layer, the DND separately projects the caption prefix against both consolidated visual streams through cross-attention, yielding two candidate textual feature updates. DNDs then employ a per-position, per-layer nomination mechanism to select between candidate updates, effectively routing the information at each decoding step according to the relevance of object-level or contextual visual cues. The dynamic routing enforces a sharper partitioning of private information across the streams and circumvents the limitations of simple feature fusion (such as addition or concatenation) (Wan et al., 19 Jan 2026).

2. Mathematical Specification

Let $d = d_{\rm model}$ (default 512). Denote $Z_r\in\mathbb R^{N_r\times d}$ as the consolidated region sequence, $Z_s\in\mathbb R^{N_s\times d}$ as the consolidated segmentation sequence, and $Z_t^l\in\mathbb R^{\mathrm{seq}\times d}$ as the text features at decoder layer $l$. Processing within a DND layer is as follows:

Masked self-attention on text:

$$\hat Z_t^l = \mathrm{LN}\Bigl(\mathrm{LN}\bigl(\mathrm{M\_MHSA}^l_t(Z_t^l,Z_t^l,Z_t^l)\bigr)+Z_t^l\Bigr) + Z_t^l$$

Cross-attention to region and segmentation:

$$M_{rt}^l = \mathrm{LN}\Bigl(\mathrm{LN}\bigl(\mathrm{MHA}_{rs}^l(\hat Z_t^l,Z_r,Z_r)\bigr) +\hat Z_t^l\Bigr) + \hat Z_t^l$$

$$Z_{tr}^l = \mathrm{LN}\bigl(\mathrm{PWFF}^l(M_{rt}^l)+M_{rt}^l\bigr)$$

$$M_{st}^l = \mathrm{LN}\Bigl(\mathrm{LN}\bigl(\mathrm{MHA}_{rs}^l(\hat Z_t^l,Z_s,Z_s)\bigr) +\hat Z_t^l\Bigr) + \hat Z_t^l$$

$$Z_{ts}^l = \mathrm{LN}\bigl(\mathrm{PWFF}^l(M_{st}^l)+M_{st}^l\bigr)$$

Dynamic Nomination:

Fuse candidate updates to form logits $\Gamma \in \mathbb R^{\mathrm{seq}\times 2}$, select for each position: $$w_i = \arg\max \left( \Gamma_{i,1}, \Gamma_{i,2} \right), \qquad i = 1 \ldots \mathrm{seq}$$ This yields a one-hot map $\Psi \in \{0,1\}^{\mathrm{seq}\times 2}$. The Gumbel-Softmax relaxation is used in training to enable gradient flow.

Update textual representation:

$$Z_t^{l+1} = Z_{tr}^l \odot \Psi_{:,0} + Z_{ts}^l \odot \Psi_{:,1}$$

Three such DND layers are stacked within DSCT.

3. Stepwise Algorithmic Description

The dynamic nomination process within a DND layer operates as follows:

H_t = M_MHSA(Z_t^l, Z_t^l, Z_t^l) H_t = LN( LN(H_t + Z_t^l) + Z_t^l ) A_r = MHA(H_t, Z_r, Z_r) A_r = LN( LN(A_r + H_t) + H_t ) Z_tr = LN( PWFF(A_r) + A_r ) A_s = MHA(H_t, Z_s, Z_s) A_s = LN( LN(A_s + H_t) + H_t ) Z_ts = LN( PWFF(A_s) + A_s ) Gamma = Linear(Z_tr + Z_ts) Psi = hard_argmax_rows(Gamma) # Differentiable Gumbel-softmax during back-prop Z_t^{l+1} = Z_tr * Psi[:,0] + Z_ts * Psi[:,1]

Note: Dropout and layer normalization are applied as described in the original architecture.

4. Architectural Specifications and Training Regimen

The DND configuration in DSCT is as follows:

• $d_{\rm model} = 512$
• Number of attention heads: 8
• Number of DND layers $L_{\mathrm{DND}} = 3$
• Number of PSMAE layers $L_{\mathrm{PSM}} = 3$
• PWFF hidden dimension $= 2048$
• Dropout keep probability: 0.9
• Batch size: 50
• Shared MHA/PWFF parameters per layer; private layer normalization parameters per stream
• Gumbel-softmax temperature annealed during training

The network is trained with the standard cross-entropy loss for captioning and later fine-tuned with self-critical sequence training using the CIDEr reward: $$L_{XE} = -\sum_{t=1}^T \log p_\theta(y^*_t \mid y^*_{<t}, I)$$

$$\nabla_\theta L_{RL} = -\frac{1}{k}\sum_{i=1}^k (r(y^{(i)})-b)\,\nabla_\theta\log p_\theta(y^{(i)}), \quad b = \frac{1}{k}\sum_i r(y^{(i)})$$

No auxiliary or stream-specific losses are introduced beyond these objectives (Wan et al., 19 Jan 2026).

5. Experimental Evaluation and Ablation Analyses

Ablation experiments quantify the distinct contribution of DND. Substituting a vanilla Transformer decoder with DND, while holding the PSMAE encoder fixed, yields performance improvements on the COCO Karpathy test split:

Method	B@1	B@4	M	R	C	S
Baseline (no PSMAE, no DND)	80.9	38.7	29.1	58.7	131.7	22.8
+ PSMAE (fusion by addition)	81.4	39.3	29.6	59.3	134.1	23.2
+ PSMAE (fusion by concat)	81.7	39.6	29.7	59.6	135.6	23.5
PSMAE + DND (full DSCT)	82.7	40.3	30.5	59.9	137.6	23.9

Transitioning from simple fusion to DND yields a gain of approximately $2.0$ CIDEr ($135.6 \to 137.6$) and $0.7$ B@4 ($39.6 \to 40.3$). Further, DND surpasses alternative fusion strategies, including multi-modal inter-attention, vector-shift attention, and iterative layer normalization, as indicated in detailed ablation tables (Wan et al., 19 Jan 2026). Qualitative analysis of the learned nomination maps reveals a tendency for DNDs to select region features for nouns or objects, and segmentation features for prepositions or relational terms, indicating semantically meaningful routing.

6. Interpretations and Implications for Multimodal Transformers

The DND module establishes a mechanism for dynamic, differentiable routing between orthogonal visual modalities conditioned on fine-grained linguistic context. This allows for adaptive bypassing of semantic inconsistencies and spatial misalignment inherent in multi-modal fusion, resulting in enhanced descriptive quality. The approach demonstrates that per-position, hard attention for multi-stream fusion is both implementable and effective within standard Transformer infrastructures. A plausible implication is that similar dynamic nomination architectures may generalize to other vision-language or multi-modal sequence generation tasks, where the optimal information source may vary temporally or contextually (Wan et al., 19 Jan 2026).