Dynamic Normalized Ensemble (DNE)

• Dynamic Normalized Ensemble (DNE) is a set of techniques that dynamically normalize and combine outputs or gradients from diverse neural models to address scale misalignment and gradient dominance.
• It leverages per-model running statistics, momentum updates, and gating networks to adaptively weight contributions during adversarial attacks and deep representation normalization.
• Empirical results demonstrate that DNE improves transferability and robustness over static averaging methods, yielding superior performance in both multi-model attacks and normalization tasks.

Dynamic Normalized Ensemble (DNE) refers to a class of techniques for dynamically combining the outputs or gradients of heterogeneous neural models or normalizers during optimization or inference. DNE mechanisms are designed to address the challenges of scale misalignment, gradient dominance, and generalization in both adversarial perturbation generation (notably in multi-model attack pipelines) and the normalization of deep representations. Key examples include the loss normalization ensemble used in dual-task adversarial speech attacks (Sun et al., 19 Jan 2026), as well as exemplar normalization for deep networks (Zhang et al., 2020). DNE enables adaptive, sample- or iteration-specific weighting of constituent models, yielding improved transferability, stability, and empirical performance versus static averaging or fixed schemes.

1. Motivation and Rationale for Dynamic Normalized Ensemble

In ensemble-based pipelines, whether for adversarial example generation or deep feature normalization, naïve averaging of loss or activation statistics can result in "gradient dominance," wherein one constituent model's scale or variance drowns out other objectives. In the DUAP adversarial framework, targeting multiple speaker recognition (SR) models with a universal perturbation often fails if a static loss formulation is used: optimization overfits to a single SR surrogate, hurting transferability to unseen models and architectures. Similarly, in feature normalization, static switchable normalization (SN) applies the same mixture weights to all samples and layers, limiting the capacity to address sample- or domain-specific covariate shifts.

The DNE strategy mitigates these challenges by dynamically normalizing ensemble members’ contributions. For adversarial attacks, this involves maintaining per-model running mean and variance estimates, normalizing instantaneous losses to zero mean and unit variance, and adaptively truncating negative deviations to focus optimization on hard-to-fool models. For deep normalization, DNE ("exemplar normalization") employs per-sample gating networks to produce customized mixture weights across several normalization methods, further reducing internal covariate shift and enhancing generalization.

2. Formalization and Mathematical Definition

DNE in Adversarial Ensemble Attack

Let $f_k$ for $k=1,\ldots,K$ denote $K$ surrogate SR models. At iteration $i$, the raw cross-entropy loss for model $k$ is

$$L_k^{raw,(i)} = -\log \frac{\exp(z_{k,s^*})}{\sum_{j=1}^J \exp(z_{k,j})},$$

where $z_{k,s^*}$ is the logit for target speaker $s^*$. Momentum-based running statistics are updated as:

$$\mu_k^{(i)} = m \mu_k^{(i-1)} + (1 - m) L_k^{raw,(i)},\ \sigma_k^{(i)} = m \sigma_k^{(i-1)} + (1 - m) (L_k^{raw,(i)})^2.$$

Standard deviation is:

$$\mathrm{std}_k^{(i)} = \sqrt{\sigma_k^{(i)} - (\mu_k^{(i)})^2 + \epsilon}.$$

Normalized loss is:

$$\tilde{L}_k^{(i)} = \frac{L_k^{raw,(i)} - \mu_k^{(i)}}{\mathrm{std}_k^{(i)}}.$$

Negative values are truncated:

$L_k^{final,(i)} = \max(0, \tilde{L}_k^{(i)}).$

Ensemble SR loss is averaged:

$$L_{SR}(\delta) = \frac{1}{K} \sum_{k=1}^K L_k^{final}.$$

DNE in Deep Representation Normalization

Let $X \in \mathbb{R}^{N \times C \times H \times W}$ be a batch input. For $K$ normalizers indexed by $k$ (e.g., BN, IN, LN), each computes statistics $(\mu^k, \delta^k)$. For each sample $n$ and layer $\ell$, per-sample mixture weights $\lambda_{\ell n}^k$ are produced by a gating network:

$$\hat{X}_n = \sum_{k=1}^K \gamma^k \left[ \lambda_n^k \frac{X_n - \mu^k}{\sqrt{(\delta^k)^2 + \epsilon}} \right] + \beta^k,$$

subject to $\lambda_n^k \geq 0,\ \sum_k \lambda_n^k = 1$. The gating function $\mathcal{G}$ operates over pooled features and pairwise correlations among normalizer channels, ensuring mixture adaptivity for each sample and each layer.

3. Algorithmic Implementation and Pseudocode

Loss Normalization-based DNE (DUAP)

delta = 0 for k in range(K): mu_k, sigma_k = 0, 0 for i in iterations: for k in range(K): z_k = f_k.embedding(x + delta) @ prototype(k, s*) L_raw_k = -log(softmax(z_k)[s*]) mu_k = m * mu_k + (1 - m) * L_raw_k sigma_k = m * sigma_k + (1 - m) * (L_raw_k ** 2) std_k = sqrt(sigma_k - mu_k ** 2 + ε) L_norm_k = (L_raw_k - mu_k) / std_k L_final_k = max(0, L_norm_k) L_SR = sum(L_final_k for k in K) / K grad = gradient(L_ASR(delta) + λ1 * L_SR + λ2 * L_psy, delta) delta = Project_inf(delta - α * sign(grad), ε) return delta

Exemplar Normalization (DNE in Deep Networks)

for k in range(K): mu_lk, delta_lk = compute_stats(X, normalizer=k) for n in range(N): lambda_ln = GatingModule(X_n, {mu_lk, delta_lk}, Θ_l) # softmax for constraints for k in range(K): Y_ln_k = (X_n - mu_lk) / sqrt((delta_lk) ** 2 + ε) X_new = sum(lambda_ln_k * gamma_lk * Y_ln_k + beta_lk for k in K)

4. Theoretical Justification and Empirical Analysis

Scale Alignment and Gradient Diversity

By normalizing each model's loss or activation scale to zero mean and unit variance, DNE ensures that gradients are not dominated by any constituent surrogate. This yields optimization directions that are approximately uniform mixtures of all objectives, rather than being skewed toward the most difficult or trivially satisfied surrogate. The truncation of negative deviations further adaptively weights the learning signal, assigning zero gradient to models already above historical mean loss.

Gradient-cosine-similarity analyses in (Sun et al., 19 Jan 2026) showed SR ensemble members have low pairwise gradient correlation, indicating uncovered subspaces and the necessity of dynamic normalization for broad manifold coverage. This mechanism underpins the improved black-box transfer to unseen architectures and tasks.

Generalization across Tasks and Benchmarks

In DUAP's speaker recognition transferability benchmarks, DNE produced SRoA-SR ≥ 0.999 across six diverse test models, including both seen (ECAPA-TDNN, WavLM, ResNet34) and unseen (X-vector, i-vector, HuBERT) architectures. In comparison, static and fixed ensemble methods collapsed on out-of-distribution surrogates, confirming the superior generalization of DNE (Sun et al., 19 Jan 2026).

For deep network normalization, EN (DNE) achieved consistent improvements across classification (ImageNet: +1.2% over SN; WebVision: +0.7%), segmentation (ADE20K: +0.5% mIoU over SN), and noisy label settings. Performance gains over SN were approximately 300% higher than SN's gain over BN on some tasks (Zhang et al., 2020).

5. Architectural Details and Constraints

Gating Network in EN for Deep Representations

The gating module for DNE is carefully constructed to maintain stability and prevent overfitting. It operates by spatial pooling of activations, pre-normalizing each normalizer's statistics, stacking pooled features, followed by 1D group convolution. A pairwise correlation matrix among normalizer activations is flattened and passed through fully connected layers with tanh activation, followed by a final softmax to enforce mixture constraints:

• Channel reduction factor $r$ (e.g., $32$) regulates parameter cost.
• Parameter counts are $C +$small FC overhead per layer.
• Sharing the same λ for mean and variance avoids destabilizing cross-terms. Pairwise correlation pooling empirically stabilizes convergence over naive MLP gating.

DNE in DUAP Adversarial Pipeline

DNE hyperparameters include momentum $m$ ($\in [0.9, 0.99]$), constant $\epsilon$ ($\sim 10^{-6}$), and ensemble size $K$ (tradeoff between diversity and computation; in practice $K=3$). The truncation of negative normalized losses is used to avoid over-correction. However, small loss variance in one surrogate can inflate its influence; further, total truncation of negative losses can momentarily slow model-wide convergence.

6. Limitations and Prospective Directions

DNE exhibits certain limitations:

• If a surrogate model exhibits minimal loss variance, its normalized loss can be artificially magnified, potentially distorting optimization signal.
• The pure truncation of negative loss deviations may delay convergence when the optimization landscape renders all surrogates under-fit.
• Fixed momentum and truncation parameters lack per-model adaptivity, possibly leaving convergence unbalanced for ensembles with stark architectural heterogeneity or dataset divergence.

Future enhancements outlined in (Sun et al., 19 Jan 2026, Zhang et al., 2020):

• Adaptive truncation thresholds to allow modest negative excursions.
• Incorporation of per-model learning rate or second-moment statistics, in analogy to optimizers like Adam.
• Application of DNE to multi-modal, multi-lingual ensembles—where model capacity and data heterogeneity diverge more dramatically.
• Theoretical investigation of variance-reduced model selection or importance-sampling dynamics for optimality in convergence and generalization.

7. Summary and Significance

Dynamic Normalized Ensemble (DNE) provides a principled framework for balancing heterogeneous objective functions or normalizer activations in both adversarial attack construction and deep neural representation learning. By implementing adaptive normalization and mixture weighting, DNE prevents optimization collapse, widens generalization, and delivers robust empirical performance in diverse tasks. Its demonstrated superiority over static ensemble methods on benchmarks and tasks with pronounced architectural or data variety underlines its relevance for scalable, transferable learning systems (Sun et al., 19 Jan 2026, Zhang et al., 2020).