Selective Negative Learning in FER

• Selective Negative Learning (SNL) is a training paradigm that mitigates bias in imbalanced facial expression datasets by penalizing overrepresented classes.
• It employs techniques such as weighted cross-entropy loss, label smoothing, and adaptive sampling to improve fairness and generalization.
• Empirical evidence shows that SNL enhances minority class accuracy and overall model performance in FER through ensemble methods and rigorous fairness metrics.

Selective Negative Learning (SNL) is a training paradigm designed to mitigate the effects of dataset bias—particularly class and demographic imbalance—by introducing mechanisms that counteract the overemphasis on majority classes or uninformative data regions. In the context of facial expression recognition (FER), where datasets such as RAF-DB and AffectNet exhibit strong skew towards particular emotions and demographic groups, SNL and related techniques play a pivotal role in improving model generalizability and fairness by selectively discouraging the model from overfitting to overrepresented categories and promoting attention to minority expressions (Hosseini et al., 16 Feb 2025, Li et al., 2019, Zhou et al., 2023).

1. Problem Motivation: Bias in Real-World FER Datasets

Imbalance in both class and demographic distributions is acute in the leading FER datasets. AffectNet, for example, contains approximately 42% "Happy" labels, while categories such as "Disgust" and "Fear" comprise only about 2% and 5% of the dataset, respectively. RAF-DB, while more balanced in some respects, still has over 31% "Anger" and significant underrepresentation of "Fear" and "Disgust." Such class skews directly degrade the ability of trivially-trained deep models to generalize, resulting in severe performance drops for rare facial expressions and high intrinsic and extrinsic dataset bias as measured by standard fairness metrics (e.g., Disparate Impact, Equalized Odds Difference) (Hosseini et al., 16 Feb 2025, Li et al., 2019).

2. Formal Definition and Theoretical Foundation

Selective Negative Learning, broadly construed, introduces mechanisms that either (a) penalize models for high confidence on common/majority classes in imbalanced regions or (b) reweight loss contributions, sample probabilities, or gradients to reduce the dominance of overrepresented groups. Formally, if the canonical loss for a sample $x_i$ with label $y_i$ is $\mathcal{L}_{CE}$ (cross-entropy), SNL strategies integrate a weighting factor or regularizer:

$$\mathcal{L}_{SNL} = w(y_i) \cdot \mathcal{L}_{CE}(x_i, y_i) + \lambda \cdot \mathcal{R}_{SNL}(f(x_i))$$

where $w(y_i)$ is inversely proportional to class frequency, and $\mathcal{R}_{SNL}$ is a selective penalty (e.g., negative logit sharpening for majority classes). In practice, these approaches generalize naive reweighting and can be realized via specialized per-class loss schedules, adaptive sampling, or adversarial objectives. The learnable class-reweighting parameter $\alpha_\ell$—used to align source and target class distributions in cross-dataset transfer—is given by (Li et al., 2019):

$$\alpha_\ell = \frac{P_{\rm tgt}(Y=\ell)}{P_{\rm src}(Y=\ell)} = \frac{\frac{1}{N_t}\sum_{i=1}^{N_t}\delta_i(\ell)}{\frac{N_s^\ell}{N_s}}$$

where $\delta_i(\ell)\approx P(\ell\mid x_i^t)$. This directly implements core SNL intuition by diminishing the loss impact from majority-class samples.

3. SNL Techniques in FER: Implementations and Variants

Within state-of-the-art FER pipelines, multiple instantiations of SNL principles are observed:

• Weighted cross-entropy loss (WCE): Directly scales the loss contribution of each sample based on inverse class frequency, most effective in extremely skewed regimes such as AffectNet where, for example, "Disgust" and "Contempt" are rare (Zhou et al., 2023).
• Label smoothing regularization (LSR): Smooths the ground-truth labels, thus discouraging overconfident predictions on dominant classes and promoting robustness against bias. RAF-DB, with moderate imbalance, benefits from combining LSR (applied to 80% of training batches) with standard cross-entropy (Zhou et al., 2023).
• Mini-batch composition: Careful sampling designs, such as stratified mini-batching or probability rescaling, ensure that minority classes are sufficiently represented in each batch, preventing gradient stochasticity from swamping rare classes.
• Learnable re-weighting in transfer: ECAN (Li et al., 2019) integrates selective negative weighting by matching both marginal and per-class conditional distributions between train and target sets, minimizing Maximum Mean Discrepancy (MMD) with class prior alignment.
• Ensemble and feature-fusion regimes: By combining networks trained with slightly different SNL-informed losses or attention masks, Top-Two Voting and feature fusion ensure minority classes can influence final predictions even in an ensemble majority-vote scenario (Zhou et al., 2023).

4. Empirical Evidence and Impact on Performance

Quantitative evaluation across RAF-DB and AffectNet demonstrates the critical efficacy of SNL-inspired approaches in both single-network and ensemble settings:

Model/Setting	RAF-DB (%)	AffectNet-8 (%)	AffectNet-7 (%)
Baseline R18	88.62	52.16	58.17
+ FAML (Multi-Loss)	90.32	62.17	65.83
+ Ensemble (T2V, 6 nets)	91.46	63.04	66.51
+ FGA & T2V (6 nets)	91.59	63.27	66.63

The combination of weighted cross-entropy or label smoothing, together with ensemble aggregation (T2V), produces absolute gains up to +1.27 percentage points on RAF-DB and stabilizes per-class accuracies for rare categories such as "Disgust" and "Fear" (improving these from mid-80% to low-90% accuracy) (Zhou et al., 2023). A plausible implication is that SNL is necessary for consistent minority-class performance when class imbalance exceeds 3–5× (Zhou et al., 2023).

5. Relationship to Fairness, Generalizability, and Transfer Learning

SNL is central to fairness-centric training in FER, as evidenced by the alignment of its goals with metrics such as Statistical Parity Difference and Equalized Odds Difference (Hosseini et al., 16 Feb 2025). By reducing overfitting to dominant demographics and emotions, SNL techniques lower both intrinsic bias (dataset-identification accuracy) and extrinsic bias (leave-one-out generalization drop). Notably, in cross-domain setups (e.g., RAF-DB $\rightarrow$ AffectNet-7), the application of class-reweighted MMD and SNL-like objectives in the ECAN model reduced domain accuracy drops from −44% to −32%, restoring performance for minority classes (Li et al., 2019).

Moreover, the modular architecture of FER pipelines employing SNL—where loss function, sampling design, and ensemble strategy are decoupled—enables principled modifications for debiasing either pre-processing, in-processing, or post-processing. These include GAN-based synthetic oversampling, fairness-regularized losses, or output calibration for subgroup error parity (Hosseini et al., 16 Feb 2025).

6. Methodological Limitations and Open Issues

Despite the effectiveness of SNL in practice, residual challenges remain:

• In severe imbalance scenarios (e.g., AffectNet "Disgust" <2% of samples), weighted loss can lead to overfitting or noise amplification if minority class samples are insufficiently diverse.
• SNL does not eliminate intra-class variation bias; "prototypical" expressions in one dataset may not generalize to subtle or culturally inflected variants in another (Li et al., 2019).
• Current SNL implementations do not natively address intersectional subgroups (e.g., age-by-gender-by-expression), requiring additional stratification or multi-head fairness objectives (Hosseini et al., 16 Feb 2025).

Future research emphasizes developing principled SNL strategies that dynamically adapt to non-stationary class priors, probabilistically estimate per-sample informativeness, and integrate adversarial debiasing for both observable and latent group attributes.

7. SNL in the Context of Model and Dataset Development

Best practices for deploying SNL in FER systems include multi-pass annotation workflows to ensure reliable ground-truths, cross-dataset benchmarking to assess generalizability, and explicit fairness reporting disaggregated by protected attributes (race, gender, age) (Hosseini et al., 16 Feb 2025). Models implementing SNL should document per-class accuracy, confusion matrix distributions, and fairness metrics such as SPD and EOD across all evaluation splits. Incorporating SNL techniques into both model design and evaluation protocols is essential for the development of robust, fair, and generalizable FER systems.