Energy Invariant Attention Fusion

• Energy Invariant Attention (EIA) is a fusion mechanism that preserves the squared Euclidean norm of combined signals, ensuring consistent energy through weighted scaling.
• It employs a decomposition-prediction-reconstruction paradigm to blend signals without amplitude distortion, improving robustness in noisy and long-horizon sequence modeling.
• EIA is applied in time series trend/seasonal fusion and Transformer multi-head attention, yielding superior performance over traditional convex fusion methods.

Energy-Invariant Attention (EIA) is a class of fusion mechanisms designed to combine multiple input signals or attention heads while preserving the total "energy"—formally the squared Euclidean (ℓ₂) norm—of the combined output. EIA addresses a key limitation in traditional convex fusion and multi-head attention approaches, where naively blending channels or heads leads to amplitude distortion and loss of signal fidelity. By enforcing energy conservation within the decomposition-prediction-reconstruction paradigm, EIA improves model stability and robustness, especially in noisy and long-horizon sequence modeling tasks. EIA has been developed and applied both as a channel-wise mechanism in trend/seasonal fusion for time series forecasting (Zhang et al., 13 Nov 2025) and as a general head-fusion framework in Transformer-style architectures (Farooq, 21 May 2025).

1. Mathematical Definition and Key Principle

EIA operates by fusing two or more feature streams such that, regardless of fusion weights, the total ℓ₂-norm of the fused output matches the norm of the direct sum. In the context of time series fusion in MDMLP-EIA (Zhang et al., 13 Nov 2025), given channel-wise predictions $y_1, y_2 \in \mathbb{R}^{Q \times C}$ (trend and seasonal components) and an adaptive fusion weight $\beta \in (0,1)^{Q\times C}$, standard weighted fusion produces

$$y_{\text{std}} = \beta \odot y_1 + (1 - \beta) \odot y_2,$$

where $\odot$ denotes elementwise multiplication. This convex combination does not, in general, conserve the overall signal energy $\|y_{\text{std}}\|_2^2$.

Energy-Invariant Attention corrects this by applying an overall scaling of $2$ to maintain invariance:

$$y_3 = 2\left[\beta \odot y_1 + (1 - \beta) \odot y_2\right].$$

When $\beta \equiv \frac{1}{2}$, this reduces to $y_3 = y_1 + y_2$. For all $\beta$, the transformation satisfies

$\|y_3\|_2^2 = \|y_1 + y_2\|_2^2,$

thereby preserving the sum-energy of the underlying components and preventing amplitude distortion.

2. Theoretical Foundation and Motivation

In decomposition-based sequence models, preserving the energy norm of the original signal through all fusion and transformation steps is theoretically justified by the need to avoid artificial magnification or attenuation of input magnitude. Weighted fusions in conventional architectures can under- or over-amplify certain components, exacerbating errors and compromising robustness, especially as prediction horizons grow (Zhang et al., 13 Nov 2025). EIA enforces

$\|y_3\|_2 = \|y_1 + y_2\|_2$

by construction, acting as an implicit regularizer that constrains the solution space, ruling out degenerate solutions that fit training objectives by distorting amplitude. In MDMLP-EIA, a proof of non-inferiority demonstrates that direct-sum fusion is a special case of EIA, and optimization from this initialization cannot increase training loss (Appendix F, Theorem 3.1 in (Zhang et al., 13 Nov 2025)).

Within multi-head attention, EIA formalizes fusion as the unique global minimizer of a convex quadratic energy functional:

$$E_{\text{fuse}}(Z) = -\sum_{h=1}^H w_h\,\mathrm{tr}(Z^\top A^{(h)}V^{(h)}) + \frac{1}{2} \left(\sum_{h=1}^H w_h\right) \mathrm{tr}(Z^\top Z),$$

yielding

$$Z^* = \frac{\sum_{h=1}^H w_h A^{(h)}V^{(h)}}{\sum_{h=1}^H w_h},$$

which is a weighted average of the single-head outputs, thus ensuring invariance across permutation, scaling, and shifting symmetries (Farooq, 21 May 2025).

3. Architectural Realization and Computational Aspects

In MDMLP-EIA (Zhang et al., 13 Nov 2025), EIA is realized as an efficient, small MLP-based attention block. The sequence of operations to compute $\beta$ is:

• Concatenate $y_1$ and $y_2$ along the channel dimension: $$Y_C = \mathrm{concat}(y_1, y_2) \in \mathbb{R}^{Q \times 2C}$$.
• Apply $$\,\texttt{Linear}_A^1\!:\,Q \times 2C \to Q \times 4C$$.
• Non-linear activation and dropout: $\texttt{GeLU}(\cdot)$, then Dropout.
• Apply $$\,\texttt{Linear}_A^2\!:\,Q \times 4C \to Q \times C$$.
• Output fusion weight: $$\beta = \operatorname{Sigmoid}(\cdot) \in (0,1)^{Q \times C}$$.
• Final fusion: $y_3 = 2[\beta \odot y_1 + (1 - \beta) \odot y_2]$.

The total parameter count is $12C^2$ (for $C$ the number of channels), which is minor relative to standard self-attention for moderate $C$. Overhead for the EIA module is $O(Q\cdot C^2)$ per forward pass.

In multi-head transformer fusion (Farooq, 21 May 2025), EIA defines the fused output as the weighted average of the per-head outputs, computed explicitly and requiring only a sum and division, without additional optimization.

4. Integration within Time Series and Attention Architectures

In MDMLP-EIA (Zhang et al., 13 Nov 2025), following RevIN normalization and exponential moving average (EMA) decomposition, EIA fuses the channel-wise trend and seasonal predictions. The output is then mapped back to the original scale via inverse normalization. This channel-level EIA ensures amplitude fidelity throughout decomposition-prediction-reconstruction, crucial for recovering weak seasonalities and for robust long-horizon inference.

Within Transformer-based sequence models, EIA provides a principled alternative to naïve summation or averaging of head outputs. Instead, EIA interprets fusion as minimization of a convex energy, leveraging the context-well perspective of Modern Hopfield Networks (Farooq, 21 May 2025). EIA thus subsumes standard multi-head averaging as a special case and enables the incorporation of head-specific weighting or learned fusion parameters, while retaining all softmax-attention invariances.

5. Empirical Results and Comparative Studies

Ablation studies in MDMLP-EIA demonstrate that EIA achieves superior predictive accuracy compared to baselines including direct sum (ADD), small MLP fusion, and adaptive gating mechanisms (AGM). Across six core datasets and prediction horizons, EIA delivers mean squared error (MSE) and mean absolute error (MAE) improvements:

• vs. ADD: $2.10\%$ lower MSE, $1.53\%$ lower MAE,
• vs. MLP: $5.91\%$ lower MSE, $5.12\%$ lower MAE,
• vs. AGM: $1.83\%$ lower MSE, $0.60\%$ lower MAE, attaining the best MSE and MAE in the majority of benchmarks (Zhang et al., 13 Nov 2025).

Qualitative evaluations on extended prediction horizons (e.g., Weather $T=720$, ETTh2 $T=720$) show that EIA maintains output stability and amplitude accuracy better than alternative fusion approaches.

Transfer experiments confirm that EIA, when integrated into independently developed frameworks (xPatch, Amplifier), reduces MSE by $1$–$2\%$ and MAE by $0.5$–$1\%$ over their native fusion mechanisms. This suggests domain-agnostic applicability and broad empirical benefit.

6. Extensions, Invariances, and Limitations

EIA supports several invariance properties inherent to attention mechanisms:

• Row-wise shift invariance in softmax normalization,
• Homogeneous scaling of queries and keys,
• Permutation invariance across fused heads or channels.

Within the transformer context, EIA fusion reduces to a convex combination, introducing no additional non-linearity beyond per-head weighting. Limitations include reliance on the learned weights to suppress contributions from inferior heads, and lack of modeling for higher-order cross-head interactions, except via energy generalization. Possible extensions involve using higher-order or non-linear Hopfield-style energies to enforce head consensus or diversity, or interpreting fusion through more complex regularized energy-minimization frameworks (Farooq, 21 May 2025).