Independent Component Analysis (ICA)

• Independent Component Analysis (ICA) is a blind source separation technique that decomposes mixed signals into statistically independent latent components based on non-Gaussianity.
• ICA employs preprocessing steps like centering and whitening, followed by optimization of independence criteria such as mutual information and kurtosis.
• Recent advancements extend ICA to noisy, time-series, sparse, and multiview applications, enhancing its effectiveness in domains like EEG analysis and image separation.

Independent Component Analysis (ICA) is a blind source separation method that models an observed random vector or multivariate time series as a linear mixture of latent statistically independent sources, typically written as $X = A S$ or, in time-indexed form, $X(t) = A S(t)$. Its central task is to estimate an unmixing matrix $W \approx A^{-1}$ such that $Y = W X$ has components that are as independent as possible. In its classical form, ICA assumes a square, invertible linear mixing matrix and non-Gaussian latent components; more recent formulations extend the framework to noisy mixtures, stationary time series, mixed spectra, multiview observations, sparsity priors, and nonparametric density estimation (Shlens, 2014).

1. Core model and identifiability

In the standard linear ICA model, the observed vector $x \in \mathbb{R}^d$ is generated as

$x = A s,$

where $s = (s_1,\dots,s_d)^T$ has statistically independent components and $A$ is an unknown invertible mixing matrix. The recovered sources are written as $\hat{s} = W(x-m)$, where $W \approx A^{-1}$ and $X(t) = A S(t)$0 is a location vector when centering is included (Spurek et al., 2017). Equivalent notation appears throughout the literature, including $X(t) = A S(t)$1 and $X(t) = A S(t)$2 for sample-indexed data (Boukouvalas et al., 2016).

ICA is identifiable only up to the standard ambiguities of permutation, sign, and scale. Permutation reflects the arbitrary ordering of recovered components; sign and scale reflect the fact that multiplying one source by a scalar and dividing the corresponding column of $X(t) = A S(t)$3 by the same scalar leaves the observed mixture unchanged. A standard identifiability condition is that at most one source is Gaussian; if all sources are Gaussian, whitening by PCA already produces a factorial distribution and ICA cannot uniquely identify a further rotation beyond the PCA basis (Shlens, 2014).

A central preprocessing step is whitening. If the centered covariance matrix is decomposed as $X(t) = A S(t)$4, then the symmetric whitening transform

$X(t) = A S(t)$5

produces data with identity covariance. Under $X(t) = A S(t)$6 and $X(t) = A S(t)$7, whitening reduces the mixing model to a rotation,

$X(t) = A S(t)$8

so the remaining ICA problem is to find an orthogonal matrix that makes the rotated coordinates maximally independent (Shlens, 2014).

2. Objective functions and statistical principles

The classical ICA objective can be formulated through mutual information. If $X(t) = A S(t)$9, then

$W \approx A^{-1}$0

and, using $W \approx A^{-1}$1, one obtains the standard ICA cost

$W \approx A^{-1}$2

where $W \approx A^{-1}$3 is constant with respect to $W \approx A^{-1}$4 (Boukouvalas et al., 2016). Minimizing this quantity is equivalent to minimizing mutual information among the recovered components.

The maximum-likelihood formulation is equivalent when the source densities are correctly specified. If the latent marginals are $W \approx A^{-1}$5, then

$W \approx A^{-1}$6

and the negative average log-likelihood differs from the mutual-information criterion by Kullback–Leibler terms measuring mismatch between the assumed and true source densities. This is the reason that accurate source-density estimation is critical in likelihood-based ICA (Boukouvalas et al., 2016).

Classical contrast functions instantiate this general principle in different ways. FastICA uses kurtosis or approximations to negentropy, often through a fixed-point iteration involving a nonquadratic nonlinearity $W \approx A^{-1}$7. Infomax maximizes output entropy and is equivalent to maximum-likelihood ICA under a suitable parametric source model. JADE uses fourth-order cumulants and joint diagonalization. These methods differ mainly in the proxy used for non-Gaussianity and in the optimization scheme, not in the underlying goal of finding a coordinate system in which the components are independent (Shlens, 2014).

3. Algorithms and computational regimes

The standard computational pipeline is centering, whitening, and optimization of an independence criterion. In one-unit FastICA, for whitened data $W \approx A^{-1}$8, a component direction $W \approx A^{-1}$9 is updated by

$Y = W X$0

followed by normalization; symmetric variants estimate all rows jointly and re-orthogonalize after each update (Shlens, 2014). Infomax instead performs gradient ascent on a log-likelihood or entropy objective, while Picard uses the same Infomax-type objective but replaces gradient descent by L-BFGS with Hessian preconditioning, yielding faster convergence in EEG benchmarks (Frank et al., 2022).

Several later methods re-engineer ICA computation by changing the optimization geometry rather than the model class. ICA-EMK uses a decoupled mutual-information cost and updates each row of $Y = W X$1 separately, which enables natural multi-core parallelization because each source index can be handled independently in the core loop (Boukouvalas et al., 2016). In a different direction, efficient estimation of unique components reformulates many-random-initialization ICA in matrix form, batches candidate runs, removes converged runs from subsequent updates, and enforces orthogonality by reducing the data rather than repeatedly orthogonalizing candidate vectors; experiments on artificial datasets and EEG data verified the efficiency of the proposed method (Matsuda et al., 2024).

This computational diversification suggests that “ICA algorithm” no longer denotes a single optimization recipe. The same independence objective can be paired with fixed-point updates, quasi-Newton methods, EM, manifold-constrained optimization, or batched matrix formulations, depending on whether the dominant difficulty is local optima, density estimation, or throughput.

4. Time-series, noise, and second-order ICA

A major limitation of classical ICA is its focus on i.i.d. samples and noiseless mixtures. For M/EEG, the noisy model

$Y = W X$2

is often more realistic than $Y = W X$3. Spectral Matching ICA (SMICA) assumes zero-mean stationary Gaussian sources and additive noise with diagonal spectral covariance, and represents the observed spectral covariance in band $Y = W X$4 as

$Y = W X$5

Under the Whittle approximation, the negative log-likelihood becomes a sum of Gaussian KL divergences between empirical and model spectral covariances, and the parameters are estimated by EM. Because $Y = W X$6 is part of the model, SMICA remains well-defined when the number of sources is smaller than the number of sensors, avoiding the degeneracy that affects noiseless ICA when $Y = W X$7 (Ablin et al., 2020).

SMICA also changes the separation principle. Independence is enforced at second order through diagonal source spectral covariances, and separation relies on spectral diversity rather than non-Gaussian instantaneous marginals. The source estimates are Wiener-filter reconstructions, which automatically down-weight noisy channels and frequencies. This makes the method directly suitable for weak-source recovery in MEG and for M/EEG artifact isolation (Ablin et al., 2020).

A more general time-series extension addresses source processes with mixed spectra. In that framework, each source is modeled as a sum of a continuous-spectrum stationary component and line-spectrum terms, and its log-spectral density is represented by cubic splines for the continuous part plus indicator functions for discrete spectral lines. The mixing matrix and mixed spectra are estimated jointly by maximizing the Whittle likelihood, and the paper proves $Y = W X$8 convergence for the unmixing matrix together with the nonparametric rate $Y = W X$9 for the estimated log-spectra under $x \in \mathbb{R}^d$0-smoothness assumptions (Lee et al., 2022).

5. Flexible densities, robustness, and structured extensions

A large branch of modern ICA replaces fixed source models by flexible or nonparametric ones. ICA-EMK estimates each source density by the maximum entropy principle under constraints given by global measuring functions and adaptive local Gaussian kernels, yielding a semiparametric exponential-family density estimate inside the ICA loop (Boukouvalas et al., 2016). Nonparametric maximum-likelihood ICA under log-concave marginals instead projects the empirical distribution onto the class of ICA distributions with log-concave marginals, and the key theoretical result is that, from the point of view of estimating the unmixing matrix, it makes no difference whether or not the log-concavity is correctly specified (Samworth et al., 2012).

Another line of work targets asymmetry rather than kurtosis. ICA based on the Split Gaussian distribution uses one-sided second moments to exploit skewness and was reported to work better than classical approaches, especially when the underlying density is not symmetric (Spurek et al., 2017). ICA based on the Split Generalized Gaussian extends this idea by allowing both asymmetry and heavy tails through left and right scales together with a shape parameter $x \in \mathbb{R}^d$1, and is designed for sources that are both skewed and heavy-tailed (Spurek et al., 2018). SparseICA-EBM augments an entropy-bound ICA objective with an $x \in \mathbb{R}^d$2-type sparsity penalty, explicitly balancing independence and sparsity in applications such as fMRI (Boukouvalas et al., 2016).

Robustness has become a distinct research direction. ICA via distance covariance minimizes a dependence measure based on a necessary and sufficient condition for independence and provides both estimation and testing procedures without assuming the existence of independent components a priori (Matteson et al., 2013). Robust ICA by robust distance correlation first applies the bowl transform, which is bounded, one-to-one, continuous, and maps far outliers to points close to the origin, and then minimizes distance correlation sequentially. The method is strongly consistent, has the usual parametric rate of convergence, and was reported to generally outperform competitors in contaminated settings (Leyder et al., 14 May 2025). Kernel-based ICA offers another high-accuracy dependence measure through kernel generalized variance or kernel canonical correlation, and randomized feature approximations reduce their computational complexity from cubic to linear in the sample size while retaining comparable separation error (Sela et al., 2016).

Structured observation models further extend ICA beyond single datasets. Multi-view ICA with shared and individual sources considers multiple views $x \in \mathbb{R}^d$3, proves identifiability of the corresponding linear structure, and introduces a model-selection procedure for recovering the number of shared sources (Pandeva et al., 2022). MVICAD adds source- and view-specific discrete delays, so that sources are shared across views up to unknown latencies, and uses alternating quasi-Newton updates for unmixing and cross-correlation-based updates for delays (Heurtebise et al., 2023). This suggests that recent ICA research increasingly treats independence as one structural prior among several, to be combined with delay, sparsity, multiview coupling, or robustness rather than imposed in isolation.

6. Applications, benchmarking, and persistent issues

ICA is a standard tool in EEG and MEG, where it is used for source separation, denoising, artifact removal, and source-subspace estimation. A comparative EEG framework evaluates ICA and other BSS algorithms using Pairwise Mutual Information (PMI), Mutual Information Reduction (MIR), and dipolarity, defined through equivalent-current-dipole fits to scalp maps. Within that framework, Picard performed similarly to Infomax ICA, whereas the more computationally complex AMICA remained best overall on MIR and dipolarity (Frank et al., 2022). In M/EEG source separation, SMICA was reported to isolate heartbeats, eye blinks, and line noise, and on phantom MEG it recovered dipole locations more precisely than usual ICA algorithms or Maxwell filtering when the dipole amplitude is low (Ablin et al., 2020).

Outside electrophysiology, ICA has long been used in the cocktail-party problem, image separation, and hyperspectral unmixing. Asymmetry-based Split Gaussian ICA and Split Generalized Gaussian ICA were evaluated on image and audio mixtures and were reported to outperform or match classical methods when the sources were skewed or heavy-tailed (Spurek et al., 2017). Robust distance-correlation ICA was illustrated on contaminated images, periodic signals, and cocktail-party data, where the bowl-based robust criterion reduced sensitivity to outliers (Leyder et al., 14 May 2025). In transcriptomics, multiview ICA with shared and individual sources recovered shared components from two large transcriptome datasets provided by two different labs and used them to obtain a plausible representation of the underlying graph structure (Pandeva et al., 2022).

Several persistent issues remain central. Non-uniqueness and local optima are intrinsic to ICA optimization and motivate many-random-initialization strategies and global-selection schemes (Matsuda et al., 2024). Model mismatch in source densities can distort gradients and degrade likelihood-based methods unless density estimation is sufficiently flexible (Boukouvalas et al., 2016). Classical noiseless ICA can become ill-posed when the number of sources is smaller than the number of sensors, forcing a PCA pre-step unless noise is modeled explicitly (Ablin et al., 2020). And a recurrent misconception is that whitening alone solves ICA: whitening removes second-order dependence, but if the latent variables are Gaussian then ICA offers no unique structure beyond PCA, whereas if they are non-Gaussian the decisive information lies in higher-order or more general dependence structure (Shlens, 2014).

Taken together, these developments portray ICA as a broad semiparametric methodology rather than a single algorithmic template. Its classical core is the search for a linear transform that minimizes dependence; its contemporary forms adapt that search to noisy spectra, shared latent structure, sparse priors, asymmetry, nonparametric densities, and robust dependence measures.