Fukunaga-Koontz Linear Discriminant Analysis

• Fukunaga-Koontz LDA is a supervised dimensionality reduction method that decorrelates within-class scatter via whitening to improve class separability.
• The method involves whitening the within-class scatter, diagonalizing the resulting between-class scatter, and constructing a closed-form projection via eigen-decomposition.
• Its practical applications include high-dimensional vision-language models, achieving significant feature compression and improved classification accuracy over traditional LDA.

Fukunaga-Koontz Linear Discriminant Analysis (FK-LDA) is a supervised dimensionality reduction technique that extends classical Linear Discriminant Analysis (LDA) by decorrelating within-class scatter through a whitening transformation before optimizing inter-class separation. Originating from statistical pattern recognition, FK-LDA identifies linear projections that both suppress within-class variation and enhance between-class discrimination. In practical contexts such as vision-LLM adaptation, FK-LDA provides a closed-form, computationally efficient procedure for reshaping embedding spaces to yield improved class separability, robust dimensionality reduction, and efficient representations for large-scale classification and retrieval tasks (Suchanek et al., 1 Feb 2026).

1. Mathematical Framework

FK-LDA considers a $D$-dimensional embedding space with $N$ labeled samples $\{(x_i, y_i)\}_{i=1}^N$, where $x_i \in \mathbb{R}^D$ and $y_i \in \{1, \ldots, K\}$.

Key definitions:

• Per-class mean: $\mu_k = \frac{1}{N_k} \sum_{i: y_i = k} x_i$
• Global mean: $\mu = \frac{1}{N} \sum_i x_i$
• Within-class scatter: $S_w = \sum_{k=1}^K S_k$, $$S_k = \sum_{i: y_i=k} (x_i - \mu_k)(x_i - \mu_k)^\top$$
• Between-class scatter: $$S_b = \sum_{k=1}^K N_k (\mu_k - \mu)(\mu_k - \mu)^\top$$

FK-LDA departs from classical LDA by transforming (whitening) the data such that $S_w$ becomes the identity, thus "sphering out" within-class variations prior to addressing between-class separation.

2. Optimization Objective

The FK-LDA objective is to maximize the Rayleigh quotient:

$$J(W) = \frac{\mathrm{tr}(W^\top S_b W)}{\mathrm{tr}(W^\top S_w W)}$$

Following whitening, $S_w \to I$, so the problem reduces to

$$J(W) = \frac{\mathrm{tr}(W^\top S'_b W)}{\mathrm{tr}(W^\top W)}$$

where $S'_b$ is the between-class scatter computed in the whitened space. Maximizing $J(W)$ yields directions of maximal class-mean separation, with within-class variation already normalized (Suchanek et al., 1 Feb 2026).

3. Closed-Form FK-LDA Solution

The procedure for FK-LDA comprises four principal steps:

1. Regularization and Diagonalization of $S_w$:
   • Form the regularized scatter $\,\bar{S}_w = S_w + \lambda I\,$, with a small $\lambda > 0$ to ensure full rank.
   • Compute $\,\bar{S}_w = V \Lambda V^\top\,$, with $$\Lambda = \mathrm{diag}(\lambda_1, \ldots, \lambda_D)$$.
   • Whitening transform: $Z = V \Lambda^{-1/2} V^\top$
2. Whitening of Class Means:
   • Whitened class mean offsets: $\,\nu_k = Z(\mu_k - \mu)$
3. Computation and Diagonalization of Whitened Between-Class Scatter:
   • $\,S'_b = \sum_{k=1}^K N_k \nu_k \nu_k^\top$
   • $\,S'_b = U \Gamma U^\top$, $\Gamma = \mathrm{diag}(\gamma_1, \ldots, \gamma_D)$
4. Final Projection Construction:
   • Retain top $L$ eigenvectors: $U_L = [u_1, \ldots, u_L]$
   • FK-LDA projection: $W = Z U_L$ (for $y = W^\top x$)

The algorithmic complexity is $O(N D^2)$ for scatter computation and $O(D^3)$ for eigendecompositions, with $O(D^2)$ storage (Suchanek et al., 1 Feb 2026).

4. Geometric and Theoretical Properties

FK-LDA's geometric effect is to "sphere" each class distribution, equalizing within-class variance in all directions. This sphered representation transforms each class into an approximately isotropic unit ball. Subsequent diagonalization of $S'_b$ yields axes along which class centroids are maximally separated, ensuring that projections onto these axes stretch inter-class distances while maintaining normalized within-class spread.

A key distinction from classical LDA is that classical LDA is constrained to $L \leq K-1$ directions (where $K$ is the number of classes), while FK-LDA, due to whitening, can produce up to $D$ directions, limited by the numerical rank of $S_w$. This suggests broader applicability in scenarios with high-dimensional, anisotropic embeddings or where the class count is large relative to embedding dimensionality (Suchanek et al., 1 Feb 2026).

5. Practical Implementation and Algorithmic Details

The FK-LDA algorithm proceeds as follows:

Step	Action	Output
1	Compute class and global means	$\mu_k,\, \mu$
2	Compute within-class scatter $S_w$ and regularize to $\bar{S}_w$	$\bar{S}_w$
3	Eigendecompose $\bar{S}_w$, derive whitening transform $Z$	$Z$
4	Compute whitened class offsets $\nu_k = Z(\mu_k - \mu)$	$\nu_k$
5	Form whitened between-class scatter $S'_b = \sum N_k \nu_k \nu_k^\top$	$S'_b$
6	Eigendecompose $S'_b$; select top $L$ eigenvectors $U_L$	$U_L$
7	Final FK-LDA projection: $W = Z U_L$	$W \in \mathbb{R}^{D \times L}$

The regularization parameter $\lambda$ is set to ensure $\bar{S}_w$ is full-rank, typically determined via cross-validation. Numerical stability can be enhanced by employing symmetric eigendecomposition for $\bar{S}_w$. For very high-dimensional data, truncated SVD or randomized eigendecomposition methods are recommended. All input data should be zero-centered before scatter computation (Suchanek et al., 1 Feb 2026).

6. Applications and Comparative Analysis

FK-LDA is particularly effective for scenarios involving high-dimensional models with anisotropic within-class covariance, such as vision-LLM embeddings (e.g., CLIP features). It has been empirically demonstrated to provide "substantial compression by up to 10-12x with little or no loss in accuracy," and to improve prototype-based classification accuracy (e.g., top-1 accuracy on ImageNet-1K improving from 75.1% to 79.1%) while supporting label spaces of up to 21K classes without degradation (Suchanek et al., 1 Feb 2026).

Compared to classical LDA, FK-LDA offers improved numerical stability—since matrix inversion for ill-conditioned $S_w$ is replaced by whitening, and its projection dimensionality is not limited to $K-1$ directions. It provides a closed-form, linear, and efficient technique for adapting and compressing large-scale feature representations in modern supervised classification pipelines (Suchanek et al., 1 Feb 2026).

7. Limitations and Theoretical Considerations

FK-LDA's effectiveness is most pronounced when within-class scatter matrices are ill-conditioned, classes have few samples, or embeddings are high-dimensional with disparate variances across directions. If within-class scatter is nearly isotropic, FK-LDA's advantage over classical LDA is diminished. Empirical robustness to regularization is observed once $\lambda$ exceeds the smallest nonzero eigenvalue of $S_w$; too small $\lambda$ may lead to instability, while excessively large $\lambda$ can attenuate discriminatory power. FK-LDA is not designed for nonlinear class boundaries, although it may facilitate subsequent nonlinear post-processing.

A plausible implication is that in large-scale applications where training data per class is limited or model representations are not naturally isotropic, FK-LDA provides a practical alternative to classical methods, prioritizing computational tractability and improved inter-class discrimination (Suchanek et al., 1 Feb 2026).