Disentanglement-Guided Loss (DGLoss)

• DGLoss is an auxiliary loss that guides the automatic separation of common and rare memory representations in prototype-based time series forecasting architectures.
• It combines separation, contrastive rarity preservation, and diversity regularization to robustly specialize memory prototypes and improve forecasting performance.
• The approach mitigates issues like pattern bleed, redundancy, and forgetting, with clearly defined loss components integrated into standard deep learning workflows.

Disentanglement-Guided Loss (DGLoss) is an auxiliary objective developed to facilitate the automatic separation and robust specialization of memory representations in prototype-based architectures for time series forecasting. Introduced within the Dual-Prototype Adaptive Disentanglement (DPAD) framework, DGLoss addresses the key failure modes associated with multi-prototype memory banks by combining pattern separation, contrastive rarity preservation, and diversity regularization. It is formulated to be model-agnostic and compatible with common deep learning workflows, improving both forecasting accuracy and representation robustness in state-of-the-art time series models (Yang et al., 23 Jan 2026).

1. Motivation and Failure Modes

DGLoss is motivated by the observation that, absent specialized regularization, learned prototype banks in deep forecasting models degrade in three significant ways:

• Pattern Bleed: Without explicit constraints, it is common for prototypes intended for rare events to encode recurring trends and vice versa, creating entanglement between “common” and “rare” memory banks.
• Redundant Collapse: Prototypes within a given bank (especially those for stable, common patterns) tend to become highly similar, reducing representational coverage.
• Forgetting of Rare Events: Rare-event prototypes, which should serve as episodic memory for infrequent but critical occurrences (e.g., anomalies, sudden regime shifts), are prone to being overwritten during training.

DGLoss counters these effects by jointly enforcing:

• Clear separation between common and rare banks,
• Memory retention for rare/critical event prototypes,
• Internal diversity among prototypes dedicated to common patterns.

2. Formal Mathematical Definition

Let $$\mathcal{P}_c = \{p_c^1, ..., p_c^M\} \subset \mathbb{R}^D$$ denote the common pattern prototypes and $$\mathcal{P}_r = \{p_r^1, ..., p_r^N\} \subset \mathbb{R}^D$$ the rare-event prototypes. For an input batch, denote the context representation by $h$. The DGLoss is composed of three terms:

• Separation Loss ($\mathcal{L}_\mathrm{sep}$), encouraging dominant activation on the correct bank (common or rare, determined by a frequency-based weight $\omega \in [0,1]$):

$$\mathcal{L}_{\mathrm{sep}} = \mathbb{E}_{\text{batch}}\left[ \omega \max(0, m - \Delta\rho) + (1-\omega)\max(0, m + \Delta\rho) \right]$$

Here, $\Delta\rho = \max(\rho_c) - \max(\rho_r)$, and $m > 0$ is a separation margin.

• Rarity Preservation Loss ($\mathcal{L}_\mathrm{rare}$), a contrastive objective within the rare bank:

$$\mathcal{L}_{\mathrm{rare}} = -\frac{1}{|\mathcal{A}|}\sum_{k \in \mathcal{A}} \log \frac {\exp(s_{k,k}/\tau)} {\sum_{j=1}^N \exp(s_{k,j}/\tau)}$$

$\mathcal{A}$ indexes rare prototypes activated for the input. $\tau$ is a temperature parameter.

• Diversity Loss ($\mathcal{L}_\mathrm{div}$), enforcing orthogonality among common prototypes:

$\mathcal{L}_{\mathrm{div}} = \frac{1}{M(M-1)} \sum_{i=1}^M\sum_{\substack{j=1\j\neq i}}^M \left(\frac{p_c^i{}^\top\,p_c^j}{\|p_c^i\|\,\|p_c^j\|}\right)^2$

The total DGLoss is a weighted sum:

$$\mathcal{L}_\mathrm{DGL} = \lambda_\mathrm{sep}\,\mathcal{L}_\mathrm{sep} + \lambda_\mathrm{rare}\,\mathcal{L}_\mathrm{rare} + \lambda_\mathrm{div}\,\mathcal{L}_\mathrm{div}$$

The overall training objective augments the forecasting loss $\mathcal{L}_\mathrm{forecast}$ (e.g., MSE):

$$\mathcal{L} = \mathcal{L}_\mathrm{forecast} + \mathcal{L}_\mathrm{DGL}$$

3. Role of Each Term and Interpretive Analysis

• $\mathcal{L}_\mathrm{sep}$ (Separation): Enforces that, for inputs estimated as “common,” the most active common-bank prototype strongly outweighs any rare-bank response (and vice versa for rare inputs). The margin $m$ is only enforced via a hinge when not already respected, resulting in adaptive regularization.
• $\mathcal{L}_\mathrm{rare}$ (Contrastive Rarity): Utilizes a batch-wise contrastive loss to maintain distinctiveness in rare-event prototypes. Activated rare prototypes are pulled closer to the context, while inactive ones are pushed away, countering prototype contamination and catastrophic forgetting.
• $\mathcal{L}_\mathrm{div}$ (Diversity): Minimizes pairwise squared cosine similarity among common-bank prototypes, maximizing coverage of the feature space over prevalent, recurring patterns.

Normalization constants ($1/|\mathcal{A}|$, $1/[M(M-1)]$) ensure sub-losses remain scale-invariant to the sizes of their respective banks.

4. Integration into Model Training

DGLoss is implemented as a drop-in auxiliary loss within the gradient update for any DPAD-enhanced forecaster. The standard workflow per optimization step is as follows:

1. Forward: Pass input $X$ through the backbone to obtain representation $h$ and compute preliminary forecast $\hat Y$.
2. Prototype Similarity: Compute similarity scores $\rho_c$ and $\rho_r$ between $h$ and all common/rare prototypes.
3. Routing and Activation: Use batch temporal statistics to set frequency-weight $\omega$ and identify active prototype indexes in rare bank $\mathcal{A}$ (commonly no more than one active rare prototype per input).
4. Loss Calculation: Assemble $\Delta\rho$, compute each loss component as described.
5. Aggregate and Backpropagate: Combine $\mathcal{L}_\mathrm{forecast}$ and $\mathcal{L}_\mathrm{DGL}$ for total loss; perform backward update for all parameters, including prototype vectors.

Pseudocode for DGLoss computation as outlined in (Yang et al., 23 Jan 2026):

for each training batch (X, Y): h = BackboneEncoder(X) Y_hat = ForecastHead(h) L_forecast = MSE(Y_hat, Y) rho_c = similarity(h, P_c) rho_r = similarity(h, P_r) delta_rho = max(rho_c) - max(rho_r) omega = frequency_weight(X) A = {k for k in range(N) if rho_r[k] > epsilon} L_sep_batch = omega * max(0, m - delta_rho) + (1 - omega) * max(0, m + delta_rho) if len(A) > 0: L_rare_terms = [-log(exp(rho_r[k]/tau) / sum(exp(rho_r[j]/tau) for j in range(N))) for k in A] L_rare_batch = mean(L_rare_terms) else: L_rare_batch = 0 L_div = sum((dot(P_c[i], P_c[j]) / (norm(P_c[i]) * norm(P_c[j]))) ** 2 for i in range(M) for j in range(M) if i != j) / (M * (M - 1)) L_DGL = lambda_sep * L_sep_batch + lambda_rare * L_rare_batch + lambda_div * L_div L_total = L_forecast + L_DGL backpropagate(L_total)

5. Hyperparameterization and Practical Advice

• Margin ($m$): Separation pressure is selected from $[0.1, 1.0]$. Increasing $m$ tightens separation but may over-penalize; smaller $m$ weakens effect.
• Temperature ($\tau$): Controls the sharpness of selection in the rare-bank contrastive loss; typical $\tau \approx 0.1$.
• Balancing Weights ($\lambda$): $\lambda_\mathrm{sep}$, $\lambda_\mathrm{rare}$, and $\lambda_\mathrm{div}$ are empirically tuned to ensure sub-losses are commensurate with the main forecasting objective. Recommended defaults are all $1.0$, with sweeps in $[0.001, 2.0]$.
• Bank sizes ($M$, $N$): Robust performance for $M \in \{32, 64, 128\}$, $N \in \{8, 16\}$. Larger prototype banks necessitate increased diversity regularization.

Optimal hyperparameter configurations are informed by monitoring validation error and prototype bank utilization. With increasing $M$ or $N$, scaling up $\lambda_\mathrm{div}$ maintains effective diversity among common prototypes.

6. Empirical Performance and Ablation Studies

Empirical results on Electricity, Weather, Traffic, and Solar benchmarks with state-of-the-art time series forecasting backbones demonstrate that DGLoss delivers reductions in Mean Squared Error (MSE) across all settings. Isolated removal of each component yields the following degradations:

Ablation	MSE (Traffic)	% change vs. full
Full DGLoss	0.416	baseline
w/o any disentanglement	0.441	+6%
w/o separation (𝓛_sep)	0.450	+8%
w/o rarity (𝓛_rare)	0.464	+12%
w/o diversity (𝓛_div, Elec)	0.177 (Elec)	+4% (Elec)

All three components are required for optimal disentanglement: separation of banks, rare-event memory, and pattern coverage. This multipronged pressure yields context-aware, robust forecasting improvements even when underlying backbone architectures and tasks vary.

7. Relationship to Broader Research Directions

DGLoss, as introduced in (Yang et al., 23 Jan 2026), is tightly integrated with prototype-based temporal memory for time series. It is distinct from approaches addressing disentanglement or gradient interference in multimodal learning via gradient path reorganization such as Disentangled Gradient Learning (DGL) (Wei et al., 14 Jul 2025), which targets optimization conflicts in multimodal models by decoupling and redirecting gradients between encoders and fusion modules. DGLoss, in contrast, is specifically targeted at enforcing memory specialization and representational coverage across temporally distributed patterns within a single modality, by means of auxiliary prototype regularization.

A plausible implication is that the principles of DGLoss—separation, contrastive memory retention, and intra-bank diversity—could generalize to memory-augmented models in other domains where prototype banks or episodic memory play a role, including sequence classification and anomaly detection.