Diversity-Augmented Losses

• Diversity-augmented losses are objective functions that integrate differentiable diversity measures into training, balancing bias, variance, and ensemble disagreement.
• They are implemented by computing per-sample losses, centroid predictions, and pairwise diversity metrics such as KL divergence or separation distances to optimize performance.
• Applications span trajectory prediction, deep metric learning, generative modeling, and class-imbalanced classification, yielding measurable improvements in robustness and accuracy.

Diversity-augmented losses are a family of objective functions and regularization terms designed to explicitly encourage or manage diversity among predictions, gradients, embedded representations, or losses within ensembles, multi-modal models, or generative networks. Rooted in the bias-variance-diversity decomposition of ensemble risk, these losses have emerged in response to the limitations of simple accuracy-based or modal aggregation objectives, especially in contexts such as deep ensemble learning, trajectory prediction, deep metric learning, generative modeling, and class-imbalanced classification. Modern frameworks treat diversity as an equal-ranking component alongside bias and variance, enabling nuanced trade-offs rather than the indiscriminate maximization of separation or disagreement.

1. General Theory: Bias–Variance–Diversity Decomposition

The central insight of diversity-augmented loss construction is that, for a wide range of losses $\ell(y,q)$, the expected risk of an ensemble according to “centroid combiner” can be decomposed into noise, average bias, average variance, and a diversity term:

$$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$

For regression losses like squared error, the expressions become fully explicit:

$$\mathbb{E}[(Y - \bar{q})^2] = \frac{1}{m}\sum_{i=1}^m (q^* - y)^2 + \frac{1}{m}\sum_{i=1}^m \mathbb{E}[(q_i - q^*)^2] - \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m (q_i - \bar{q})^2\right]$$

The diversity term is negative in the decomposition, directly reducing expected risk for fixed bias and variance. However, diversity can increase bias or variance: thus, optimalization involves managing—not unilaterally maximizing—that trade-off (Wood et al., 2023).

2. Construction and Implementation of Diversity-Augmented Objectives

Diversity-augmented losses are built by injecting a differentiable diversity measure $D(\{\theta_j\})$ into the empirical risk minimization objective, controlled by a weight $\lambda$:

$$J(\theta_1, \ldots, \theta_m) = \frac{1}{m} \sum_{j=1}^m R(\theta_j) - \lambda D(\{\theta_j\})$$

where $R(\theta_j)$ is the usual risk and $D$ estimates the population diversity term. For practical training, especially in deep learning, the diversity term is computed per minibatch, typically as the average separation between model outputs (or embeddings, gradients, or trajectories).

A representative workflow involves:

• Computing per-member losses across the batch.
• Computing the centroid or mean prediction.
• Calculating diversity across ensemble members or modes.
• Backpropagating the combined loss, tuning $\lambda$ via cross-validation to balance accuracy, robustness, and diversity.

Diversity terms can take multiple forms: average pairwise separation, Kullback–Leibler divergences, normalized differences, or domain-specific indicators (e.g., on-road feasibility) (Wood et al., 2023, Bui et al., 2024, Rahimi et al., 2024). Implementation typically leverages vectorization for computational efficiency (see pseudocode blocks in (Rahimi et al., 2024, Wood et al., 2023)), and often utilizes feasibility masks, normalization, or scaling to prevent pathological model behavior.

3. Diversity Losses in Specific Domains

a. Vehicle Trajectory Prediction

The Mode Diversity loss for multi-modal trajectory prediction encourages multiple feasible predicted trajectories to be well separated:

$L_\text{diversity}(\mathbf{y}) = \sum_{i=1}^M\sum_{j=i+1}^M \mathds{1}(i)\mathds{1}(j) \frac{1}{T}\sum_{t=1}^T \|\mathbf{y}^i_t - \mathbf{y}^j_t\|_2$

Here, only modes inside the drivable area contribute. Diversity is balanced with off-road and directional consistency losses, with hyperparameters $$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$0 separately swept and jointly fine-tuned (Rahimi et al., 2024). Empirical ablations show substantial gains in coverage and diversity metrics with minimal impact on main accuracy metrics.

b. Deep Metric Learning

In “Ensemble of Loss Functions to Improve Generalizability of Deep Metric Learning methods,” diversity is enforced by combining multiple loss functions, with an optional spread-out regularizer:

$$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$1

where $$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$2 is an average pairwise distance across normalized embeddings from distinct heads (Zabihzadeh et al., 2021). Joint optimization pushes the feature extractor toward more transferable representations, yielding superior recall and clustering on standard benchmarks.

c. Generative Modeling

Normalized Diversification penalizes generators for producing output samples with lower-than-expected normalized pairwise distances compared to latent space:

$$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$3

Hence, outputs are forced to unfold according to the geometry of their inputs, mitigating mode collapse (Liu et al., 2019).

d. Sharpness Minimization in Ensembles

DASH combines standard empirical risk, explicit sharpness penalties, and a cross-KL diversity penalty:

$$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$4

with explicit theoretical and empirical support for boosting generalization, robustness, and calibrated uncertainty (Bui et al., 2024).

4. Diversity-Augmented Losses and Class Imbalance

Cardinality-augmented losses introduce diversity invariants (magnitude and spread) from mathematical metric space theory, effectively aggregating per-sample losses through their effective diversity:

• Magnitude-based loss:

$$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$5

• Spread:

$$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$6

These losses amplify minority-class distinctions, yielding improved F1-macro and PR–AUC on both synthetic and real-world class-imbalanced datasets (O'Malley, 8 Jan 2026).

5. Theoretical Foundations and Generalizations

Generalized ambiguity decompositions, as in Audhkhasi et al., extend diversity-based risk decompositions to arbitrary twice-differentiable losses and convex ensembles:

$$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$7

with $$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$8 being a loss-adaptive diversity term and $$\mathbb{E}[\ell(Y, \bar{q})] = \text{noise} + \text{average bias} + \text{average variance} - \text{diversity}$$9 a curvature spread correction. For convex losses the bound is tight, providing a robust rationale for diversity-augmented empirical optimization (Audhkhasi et al., 2013).

6. Tuning Diversity Regularization and Empirical Trade-offs

Hyperparameters governing diversity penalties (e.g., $$\mathbb{E}[(Y - \bar{q})^2] = \frac{1}{m}\sum_{i=1}^m (q^* - y)^2 + \frac{1}{m}\sum_{i=1}^m \mathbb{E}[(q_i - q^*)^2] - \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m (q_i - \bar{q})^2\right]$$0, $$\mathbb{E}[(Y - \bar{q})^2] = \frac{1}{m}\sum_{i=1}^m (q^* - y)^2 + \frac{1}{m}\sum_{i=1}^m \mathbb{E}[(q_i - q^*)^2] - \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m (q_i - \bar{q})^2\right]$$1, $$\mathbb{E}[(Y - \bar{q})^2] = \frac{1}{m}\sum_{i=1}^m (q^* - y)^2 + \frac{1}{m}\sum_{i=1}^m \mathbb{E}[(q_i - q^*)^2] - \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m (q_i - \bar{q})^2\right]$$2) are typically optimized via cross-validation. Too much diversity can harm mean accuracy (by artificially decorrelating or biasing member predictions), while too little allows collapse to homogeneous ensembles or mode collapse in generative networks. Empirical studies confirm optimal regimes:

Task	Diversity-up Loss Weight	Typical Gains
Trajectory (Wayformer, Argoverse 2)	$$\mathbb{E}[(Y - \bar{q})^2] = \frac{1}{m}\sum_{i=1}^m (q^* - y)^2 + \frac{1}{m}\sum_{i=1}^m \mathbb{E}[(q_i - q^*)^2] - \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m (q_i - \bar{q})^2\right]$$3	+12–30% diversity metric
Metric Learning (WEDL-DML)	$$\mathbb{E}[(Y - \bar{q})^2] = \frac{1}{m}\sum_{i=1}^m (q^* - y)^2 + \frac{1}{m}\sum_{i=1}^m \mathbb{E}[(q_i - q^*)^2] - \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m (q_i - \bar{q})^2\right]$$4	+7–10% Recall@1
GAN (ndiv)	$$\mathbb{E}[(Y - \bar{q})^2] = \frac{1}{m}\sum_{i=1}^m (q^* - y)^2 + \frac{1}{m}\sum_{i=1}^m \mathbb{E}[(q_i - q^*)^2] - \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m (q_i - \bar{q})^2\right]$$5	+33% mode coverage
Classification (Cardinality Mag)	n/a	+5–10% F1-macro
Ensemble (DASH)	$$\mathbb{E}[(Y - \bar{q})^2] = \frac{1}{m}\sum_{i=1}^m (q^* - y)^2 + \frac{1}{m}\sum_{i=1}^m \mathbb{E}[(q_i - q^*)^2] - \mathbb{E}\left[\frac{1}{m}\sum_{i=1}^m (q_i - \bar{q})^2\right]$$6	+2–7% ensemble accuracy

Over-penalization can lead to loss of feasibility (as in trajectory domains) or reduction in precision (as in cardinality-based aggregation for class imbalance). Diversity-regularization is most effective with base models prone to overfitting/high variance, in homogeneous ensembles, and with limited data (Wood et al., 2023, Rahimi et al., 2024).

7. Applications and Limitations

Diversity-augmented loss frameworks apply broadly:

• All forms of ensemble learning where correlated errors degrade fusion performance.
• Multi-modal or multi-hypothesis predictors (e.g., path planning, generative models) that must cover a range of plausible futures.
• Deep metric learning generalized to zero-shot retrieval.
• Class-imbalanced learning (cardinality invariants).
• Models requiring adversarial robustness (gradient decorrelation).

Limitations include increased computational cost (due to pairwise calculations and/or cross-model gradients), sensitivity in tuning diversity weights, occasional loss of semantic diversity (when separation is enforced in Euclidean or metric space only), and practical need for feasibility or normalization filters (Liu et al., 2019, Rahimi et al., 2024).

• “A Unified Theory of Diversity in Ensemble Learning” (Wood et al., 2023)
• “A Multi-Loss Strategy for Vehicle Trajectory Prediction: Combining Off-Road, Diversity, and Directional Consistency Losses” (Rahimi et al., 2024)
• “Ensemble of Loss Functions to Improve Generalizability of Deep Metric Learning methods” (Zabihzadeh et al., 2021)
• “Generalized Ambiguity Decomposition for Understanding Ensemble Diversity” (Audhkhasi et al., 2013)
• “Normalized Diversification” (Liu et al., 2019)
• “Cardinality augmented loss functions” (O'Malley, 8 Jan 2026)
• “Diversity-Aware Agnostic Ensemble of Sharpness Minimizers” (Bui et al., 2024)
• “Improving Adversarial Robustness of Ensembles with Diversity Training” (Kariyappa et al., 2019)