Empirical Covariance (EC) Model

• The EC model is a rigorous mathematical framework that defines empirical covariance matrices and characterizes their spectral properties under high-dimensional limits.
• It employs a joint large-scale limit and Fourier transform-based scaling laws to reveal universal spectral fluctuations and phase transition phenomena.
• The model underpins practical applications such as AR process analysis and perturbation expansions, while highlighting assumptions and limitations in finite-sample scenarios.

The Empirical Covariance (EC) model provides a rigorous mathematical framework for characterizing the statistical fluctuations of empirical covariance (and auto-covariance) matrices derived from stationary stochastic processes, especially in high-dimensional and large-sample regimes. The central object is the empirical auto-covariance matrix constructed from a set of observations of a stationary process. The EC model enables analytical and semi-analytical predictions for the spectral properties of these matrices in joint scaling limits, elucidates the universality of spectral fluctuations, and forms the basis for precise perturbation expansions and concentration inequalities for empirical eigenvalues and eigenvectors. Its underlying universal features, phase transition phenomena, and connections to classical random matrix ensembles make it a key theoretical structure in modern high-dimensional statistics and stochastic process analysis (Kuehn et al., 2011, Jirak et al., 2018).

1. Construction of the Empirical Covariance Matrix

Given a real, zero-mean, second-order stationary time series $\{x_t\}_{t\in\mathbb{Z}}$, the empirical auto-covariance matrix $C$ of size $N\times N$ is constructed from $M$ consecutive observations $x_1,\dots,x_{M+N}$ as

$$C_{i j} = \frac{1}{M} \sum_{t=1}^M x_{t+i}x_{t+j},\qquad i,j=1,\dots,N.$$

By stationarity of the underlying process, the true (population) auto-covariance depends only on the lag: $\bar C(k)=\mathbb{E}[x_t x_{t+k}]$. The empirical matrix $C$ inherits an approximate Toeplitz structure, which is perturbed by finite-sample fluctuations. In the context of abstract Hilbert space analysis, for a mean-zero random element $X$ in a separable Hilbert space $\mathcal{H}$, the population covariance $\Sigma$ and its empirical counterpart $\hat\Sigma$ are

$$\Sigma = \sum_{j\ge1}\lambda_j\,u_j\otimes u_j; \qquad \hat\Sigma = \frac1n\sum_{i=1}^n X_i\otimes X_i = \sum_{j\ge1} \hat\lambda_j\,\hat u_j\otimes \hat u_j,$$

where $\{X_i\}$ are i.i.d. draws (Kuehn et al., 2011, Jirak et al., 2018).

2. Spectral Density in the Joint Limit: Scaling Law and Universality

The EC model is fundamentally concerned with the large-size regime $N\to\infty$, $M\to\infty$, while keeping the aspect ratio $\alpha=N/M\in(0,1)$ fixed. In this thermodynamic limit, the eigenvalue distribution of $C$ does not collapse (as would occur for $M\gg N^2$) nor trivialize (as for $N$ fixed, $M\to\infty$). The limiting spectral density $\rho(\lambda)$ is governed by both the true covariance structure and sampling noise. Letting $\hat C(q)$ be the Fourier transform ("symbol") of the population auto-covariance: $$\hat C(q) = \sum_{k=-\infty}^{\infty} \bar C(k) e^{iqk}, \qquad q\in[0,\pi],$$ the limiting spectral density admits the scaling representation

$$\rho(\lambda) = \int_0^{\pi} \frac{dq}{\pi} \frac{1}{\hat C(q)}\,\rho^{(0)}_\alpha\left( \frac{\lambda}{\hat C(q)} \right).$$

Here, $\rho^{(0)}_\alpha$ is the "universal" spectral density corresponding to the i.i.d. case, i.e., $\hat C(q)\equiv 1$ (Kuehn et al., 2011).

The universality of $\rho^{(0)}_\alpha$ (independence from higher moments of the i.i.d. distribution, contingent on finite variance) is analogous to Marčenko–Pastur universality for Wishart matrices. The scaling law integrates over all local variances in the frequency domain, corresponding to the Toeplitz structure in the large-$N$ Szegő limit.

3. Universal Spectral Law and Explicit Formulas

The universal spectral law for the uncorrelated (i.i.d.) case is given by an explicit closed form: $$I_\alpha(x) = \int_0^\infty dy\,e^{-ixy}(1-iy)^{-2/\alpha} = i(-x)^{-1+2/\alpha} e^{-x} \Gamma \left(1-\frac{2}{\alpha}, -x \right),\quad \Im x<0,$$ with $\Gamma(s,z)$ the incomplete gamma function. The spectral density is then

$$\rho^{(0)}_{\alpha}(\lambda) = -\frac{1}{\pi} \lim_{\varepsilon\to 0^+} \Im \frac{\partial}{\partial\lambda} \ln I_{\alpha} \left( \frac{2}{\alpha} (\lambda - i\varepsilon) \right).$$

Although $\rho^{(0)}_{\alpha}$ depends nontrivially on the aspect ratio $\alpha$, it does not depend on higher-order moments of the innovation process, only its variance. This precise closed-form facilitates both numerical evaluations and analytic studies of spectral properties in high-dimensional regimes (Kuehn et al., 2011).

4. Application to Auto-Regressive Processes and Numerical Methods

Auto-regressive (AR) processes exemplify analytically tractable settings for the EC model. For an AR($p$) process,

$$x_n + \sum_{k=1}^p a_k x_{n-k} = \sigma \xi_n,\quad \xi_n \sim \text{i.i.d.}(0,1),$$

the population spectral density is given by

$$\hat C(q) = \sigma^2 \left|1 + \sum_{k=1}^p a_k e^{-iqk}\right|^{-2},$$

yielding

$$\rho_{\text{AR}(p)}(\lambda) = \int_0^\pi \frac{dq}{\pi} \frac{|1+\sum a_k e^{-iqk}|^2}{\sigma^2} \, \rho^{(0)}_\alpha \left( \frac{ \lambda |1+\sum a_k e^{-iqk}|^2 }{ \sigma^2 } \right).$$

Numerically, this integral is discretized over $q$, with $\rho^{(0)}_\alpha$ evaluated as above. These predictions agree closely with simulations for AR processes for a broad range of $\alpha$ (Kuehn et al., 2011).

5. Perturbation Theory and Concentration Inequalities for EC Operators

For finite or infinite-dimensional Hilbert spaces, the EC model's perturbative analysis provides sharp first-order expansions and concentration results for empirical eigenvalues $\hat\lambda_j$ and spectral projectors. Introducing the relative rank

$$r_j(\Sigma) = \sum_{k\ne j} \frac{\lambda_k}{|\lambda_j - \lambda_k|} + \frac{\lambda_j}{g_j}, \quad g_j = \min(\lambda_{j-1} - \lambda_j, \lambda_j - \lambda_{j+1}),$$

the proximity and separation of population eigenvalues are quantified. Under suitable coefficient bounds, for simple $\lambda_j$ and small $x$,

$$|\hat\lambda_j - \lambda_j - \lambda_j \bar\eta_{jj}| \le C x^2 r_j(\Sigma) \lambda_j,$$

with $$\bar\eta_{kl} = \langle u_k, (\hat\Sigma - \Sigma) u_l \rangle / \sqrt{\lambda_k \lambda_l}$$. The first-order expansion for empirical eigenvectors and projectors similarly involves $r_j(\Sigma)$ and related quadratic forms (Jirak et al., 2018).

For higher-order multiplicities (degenerate eigenvalues), block expansions are available, and concentration bounds such as

$$\frac{|\hat\lambda_j - \lambda_j|}{\lambda_j} \le C_1 \sqrt{ \frac{ \log n }{ n } }$$

hold with high probability when $r_j(\Sigma) \ll n/\log n$ and $p > 4$ moments exist.

6. Phase Transitions, Consistency, and Limit Theorems

The perturbation theory for EC models exhibits sharp phase transitions for eigenvalue and eigenspace consistency, governed by the relative rank $r_j(\Sigma)$. When $r_j(\Sigma) \ll \sqrt n$, classical central limit theorems (CLT) and small error expansions hold for empirical eigenvalues and eigenspaces. Violation of this barrier ($r_j(\Sigma)/\sqrt n \not\to 0$) induces inconsistency: empirical eigenvectors can become asymptotically orthogonal to their population counterparts, and upward one-sided bias appears for non-leading directions (Jirak et al., 2018). These phenomena extend to settings with dependent or long-range dependent data under appropriate probabilistic moment conditions.

7. Assumptions, Limitations, and Extensions

Critical assumptions underlying the EC model include absolute summability of the true auto-covariance (ensuring the Fourier transform's existence), the joint large-$N$, large-$M$ scaling, and applicability of the central limit theorem for relevant overlaps. The analytic derivation leverages Szegő's theorem for Toeplitz matrices and annealed, replica-symmetric evaluation of the resolvent. Sub-leading corrections in $1/N$ or corrections at large $\alpha$ are not included. Neglect of correlations among intermediate integration variables, though empirically justified, represents a technical simplification. Extensions are possible to contexts such as heavy-tailed innovations (via generalized limit laws), cross-covariance matrices for multivariate time series, locally stationary or nonstationary processes, and to corrections at finite sample sizes using $R$-transform methods (Kuehn et al., 2011).

A summary of modeling assumptions and their roles is provided:

Assumption	Role	Limitation
Absolute summability of $\bar C(k)$	Ensures Fourier transform $\hat C(q)$	Excludes some long-range processes
Finite second moment	Justifies universality of $\rho^{(0)}_\alpha$	Excludes heavy-tailed without modification
$N,M\to\infty$, $\alpha=N/M$ fixed	Validates limiting spectral density	Finite-size effects neglected
CLT for overlaps	Underpins scaling relation	May not hold for heavy tails

These foundational constraints indicate that while the EC model yields a mathematically robust and predictive theory for empirical spectra, care must be taken when extrapolating to non-stationary, heavy-tailed, or strongly dependent data scenarios. A plausible implication is that ongoing research could extend the core EC machinery to cover broader process classes using generalized probabilistic and spectral tools (Kuehn et al., 2011, Jirak et al., 2018).