MeanDiff Classifier: Fairness & Discrimination

• MeanDiff classifier is a method that regularizes mean differences in model outputs across groups, using techniques like kernel-based MMD for fairness enforcement.
• It balances classification accuracy with subgroup fairness by integrating a discrepancy penalty into the loss, reducing disparities such as false positive rate gaps.
• The matrix-valued LDA variant introduces sparsity in pairwise class mean differences to enhance interpretability and discriminative power in multiclass settings.

The MeanDiff classifier is a principled approach for learning models that explicitly regularize certain differences in means of model outputs across groups or classes. The term refers to two related but distinct formulations in the literature: (1) a kernel-based distribution matching framework for fairness regularization in general classifiers, most notably as the MinDiff classifier for enforcing equalized false positive rates, and (2) a penalized likelihood approach for multiclass discriminant analysis with matrix-valued predictors that encourages sparsity in pairwise class mean differences. Both leverage mean-difference penalties to improve interpretability, statistical efficiency, and—for applied prediction—balance performance on primary and auxiliary metrics such as subgroup fairness or discriminative power.

1. Optimization Objective and Formalism

Kernel-Based MinDiff/MeanDiff Framework

The MinDiff (MeanDiff) classifier is trained to minimize an objective combining the standard classification loss with a statistical discrepancy regularizer that measures differences in predicted scores between subgroups. Let $f(x;\theta)$ be the model with parameters $\theta$, and consider a sensitive attribute $A \in \{0,1\}$. The training loss is

$$L(\theta) = \ell_{\text{cls}}(\theta) + \lambda \cdot D(P_0, P_1; \theta)$$

where

• $$\ell_{\text{cls}}(\theta) = \mathbb{E}_{(x, y)}[\ell_{\text{primary}}(f(x; \theta), y)]$$ is the average classification loss,
• $P_0$ and $P_1$ are the empirical distributions of model scores for subgroups $A = 0$ and $A = 1$ (typically restricted to negative examples for equality of opportunity),
• $D(\cdot, \cdot;\theta)$ quantifies the distributional discrepancy (selected below as Maximum Mean Discrepancy, MMD),
• $\lambda \geq 0$ trades off accuracy and fairness (Prost et al., 2019).

Matrix-Valued LDA MeanDiff Variant

Given $(X_i, y_i)$ with $X_i \in \mathbb{R}^{r \times c}$ and $y_i \in \{1, \dots, K\}$, the model estimates class-specific mean matrices $\mu_k$ and a Kronecker-structured covariance $\Sigma = \Delta^{-1} \otimes \Phi^{-1}$, seeking to minimize the penalized negative log-likelihood:

$$(\hat{\mu}, \hat{\Phi}, \hat{\Delta}) = \arg\min_{\mu, \Phi, \Delta} \left\{ g(\mu, \Phi, \Delta) + \lambda_1 \sum_{j<k} \|w_{j,k} \circ (\mu_j - \mu_k)\|_1 + \lambda_2 \|\Delta \otimes \Phi\|_1 \right\}$$

subject to $\|\Phi\|_1 = r$ (Molstad et al., 2016). This mean-difference penalty fuses entries of $\mu_j$ and $\mu_k$, producing sparsity in mean differences across classes.

2. MeanDiff via Kernel-Based Distribution Matching

The central innovation in the kernel-based formulation is to apply the Maximum Mean Discrepancy (MMD) in an RKHS to penalize distributional divergence between score distributions across sensitive groups:

$$\mathrm{MMD}^2(P_0, P_1) = \left\| \mu_{P_0} - \mu_{P_1} \right\|_{\mathcal{H}}^2 = \mathbb{E}_{x,x' \sim P_0}[k(x,x')] + \mathbb{E}_{x,x' \sim P_1}[k(x, x')] - 2 \mathbb{E}_{x \sim P_0, x' \sim P_1}[k(x,x')]$$

Empirically, for samples $S_0$ from $P_0$ and $S_1$ from $P_1$, an unbiased estimator is constructed using Gram matrices of the kernel function.

This procedure enables nonparametric matching of score distributions, crucial in addressing fairness metrics such as equalized false positive rates where only conditioning on certain labels (e.g., $Y=0$) is appropriate.

3. Algorithms and Implementation

MinDiff/MMD Classifier Training

The MinDiff classifier integrates the MMD penalty into minibatch stochastic optimization. Each iteration includes:

• Sampling minibatches and splitting into slices by sensitive attribute and (optionally) by label,
• Forward pass to compute scores,
• Construction of kernel Gram matrices for slices $S_0$ and $S_1$,
• Calculation of empirical MMD$^2$,
• Combination of classification and discrepancy gradients,
• Parameter update.

Hyperparameters include the kernel form (Gaussian or Laplace), kernel bandwidth, and the regularization trade-off $\lambda$. Computational cost is $O(B^2)$ kernel evaluations per batch, manageable for moderate batch sizes on GPU/TPU architectures (Prost et al., 2019).

Matrix-LDA MeanDiff Estimator

Optimization proceeds via block coordinate descent across means, precision matrices, and Kronecker factors. The means are updated using an alternating-minimization algorithm (soft-thresholded difference updates via augmented Lagrangian), and precision matrices via graphical lasso with normalization to preserve identifiability. Iterative cycling continues until the penalized likelihood converges (Molstad et al., 2016).

4. Practical Considerations and Stabilization

Several interventions are proposed to ensure stable and effective training for the MinDiff classifier:

• Warm-starting: Initial training with only the classification loss, gradually introducing the fairness regularizer by ramping up $\lambda$.
• Gradient clipping of the MMD component if it dominates early gradients.
• Prefer squared MMD penalty (avoiding the square-root) for numerical stability.
• Kernel bandwidth selection is heuristic, typically set proportional to the standard deviation of model scores. Empirical studies recommend lengthscales in $[10^{-2}, 10^1]$, with adjustments based on observed score distributions (Prost et al., 2019).
• For large batches, random Fourier features may be used to approximate kernel evaluations.

In the matrix-LDA setting, Nesterov-type momentum and periodic restarts are applied to the alternating-minimization scheme to accelerate convergence (Molstad et al., 2016).

5. Interpretation and Empirical Performance

The mean-difference/fusion penalties enforce model sparsity: in matrix-LDA, many entries of $\hat{\mu}_j - \hat{\mu}_k$ are exactly zero, highlighting only the “active” features responsible for class discrimination. Similarly, in MinDiff classifiers, penalizing MMD between subgroup score distributions selects for models in which predictions are more equitably distributed across sensitive groups.

Empirical studies demonstrate:

• On the UCI Adult dataset: MMD-based MinDiff reduces subgroup FPR_gap below 0.01 while maintaining $>83\%$ accuracy, a regime in which correlation-based penalties fail to reduce bias as sharply without degrading performance (Prost et al., 2019).
• In industrial classifiers and recommender systems, MMD-based MinDiff achieves superior subgroup fairness (e.g., 45% improvement in FPR ratio gap over correlation-based penalties) with negligible main-task performance loss.
• In EEG matrix-LDA applications, the MeanDiff estimator achieves high accuracy (98/122 leave-one-out) and improved interpretability, identifying discriminative time-channel pairs between population groups (Molstad et al., 2016).

6. Pseudocode Illustration

A high-level pseudocode structure for the MinDiff classifier is as follows:

Initialize θ for each minibatch B = sample(D, B): # Forward pass S = {f(x_i; θ) for i in B} cls_loss = sum(ℓ_primary(S[i], y_i) for i in B) # Build negative-example slices S0 = [S[i] for i in B if A_i == 0 and y_i == 0] S1 = [S[j] for j in B if A_j == 1 and y_j == 0] n0 = len(S0) n1 = len(S1) if n0 < 2 or n1 < 2: mmd_loss = 0 else: # Compute unbiased MMD^2 K00 = pairwise_kernel(S0, S0, k, ℓ) K11 = pairwise_kernel(S1, S1, k, ℓ) K01 = pairwise_kernel(S0, S1, k, ℓ) sum00 = (sum K00[i, j] for i != j) / (n0 * (n0 - 1)) sum11 = (sum K11[i, j] for i != j) / (n1 * (n1 - 1)) sum01 = (sum K01[i, j]) / (n0 * n1) mmd_sq = sum00 + sum11 - 2 * sum01 mmd_loss = mmd_sq # Backward pass total_loss = cls_loss + λ * mmd_loss θ_grad = grad(total_loss, θ) θ = θ - η * θ_grad return θ

(Prost et al., 2019)

This pseudocode encapsulates the core MeanDiff paradigm for fairness-driven regularization in machine learning.

7. Related Methods and Connections

The MeanDiff paradigm is contrasted with correlation-based penalty functions (“Corr”), which directly penalize linear score differences across groups. Empirical results underscore the superior Pareto optimality of MMD-based MinDiff at balancing subgroup fairness and overall task performance.

In the context of discriminant analysis, the MeanDiff classifier is closely related to fused means and graphical lasso regularization frameworks. Blockwise coordinate descent and alternating minimization algorithms exploit the problem structure, particularly Kronecker factorizations, to achieve statistical and computational efficiency.

The kernel-based and matrix-valued MeanDiff approaches exemplify the methodological trend of integrating interpretability, regularization, and distributional alignment within supervised learning, with direct implications for both statistical fairness and discriminative accuracy across real-world applications (Prost et al., 2019, Molstad et al., 2016).