Orthogonal Decomposition of the Tangent Space

• Orthogonal decomposition of the tangent space partitions a statistical manifold into a covariate subspace and its orthogonal residual, enabling finite-dimensional Fisher information analysis.
• The covariate Fisher Information Matrix (cFIM) is derived by restricting the Fisher-Rao metric to the covariate subspace, bridging geometric, probabilistic, and statistical inference.
• This framework underpins variance bounds, efficient score computations, and intrinsic dimensionality tests, enhancing inference in autoregressive and error-prone regression models.

The Covariate Fisher Information Matrix (cFIM) is a formalism that generalizes and renders tractable the Fisher information in a range of statistical and information-geometric contexts where covariates are either observed with error, endogenous, or serve as the primary locus of meaningful variation. It plays a critical role in nonparametric information geometry, autoregressive modeling, and error-prone regression scenarios, offering both a finite-dimensional representative of information theoretic quantities and a concrete bridge between geometric, probabilistic, and statistical inference frameworks (Cheng et al., 25 Dec 2025, Gao et al., 2017, Heavens et al., 2014).

1. Foundational Definition and Geometric Framework

The cFIM arises from an orthogonal decomposition of the tangent space $T_fM$ of an infinite-dimensional nonparametric statistical manifold $M$, where elements are smooth, positive probability densities $f$ on $\mathbb R^n$. The tangent space to $M$ at $f$ is given by

$$T_fM = \{h \in C^\infty(\mathbb R^n) : \int h(x)\, dx = 0\}$$

with the Fisher-Rao metric functional

$$g_f(h_1, h_2) = \int_{\mathbb R^n} \frac{h_1(x) h_2(x)}{f(x)}\, dx.$$

Choosing observed coordinates $x = (x_1, ..., x_n)$, one defines the covariate subspace $S \subset T_fM$ as

$$S = \mathrm{span}\Bigl\{ \partial_i f : i = 1, ..., n \Bigr\},$$

with $\partial_i f = \frac{\partial f}{\partial x_i}$. This leads to an orthogonal decomposition

$T_fM = S \oplus S^\perp,$

where $S^\perp$ is the residual subspace orthogonal to $S$. Every $h \in T_fM$ has a unique decomposition $h = h_S + \varepsilon$ with $h_S \in S$, $\varepsilon \in S^\perp$ (Cheng et al., 25 Dec 2025).

2. cFIM: Finite-Dimensional Realization and Statistical Properties

Restricting the Fisher-Rao metric to $S$, one defines the cFIM $G_f$ as

$$(G_f)_{ij} = g_f(\partial_i f, \partial_j f) = \mathbb E_{X \sim f}[s_i(X) s_j(X)]$$

where $s_i(x) = \partial_i \ln f(x)$ are the coordinate score functions. In matrix form,

$$G_f = \left( \mathbb E_f[\partial_i \ln f \, \partial_j \ln f] \right)_{i,j=1}^n.$$

Under mild regularity, $G_f$ is positive-definite and invertible. It serves as a computable, finite-dimensional approximation to the infinite-dimensional Fisher-Rao metric, capturing the total Fisher information available in the observed covariates.

A key result is the Trace Theorem:

$$H_G(f) := \mathbb E_f[\| \nabla \ln f(X) \|^2] = \mathrm{Tr}(G_f),$$

where $H_G(f)$, known as G-entropy, quantifies the total explainable information from the observed covariates (Cheng et al., 25 Dec 2025).

3. Connections to KL Divergence and Variance Bounds

The cFIM is linked to the curvature of the Kullback-Leibler divergence:

$$g_f(h, h) = \left.\frac{d^2}{dt^2}\right|_{t=0} D_{\mathrm{KL}}(f \| f_t)$$

for a smooth path $f_t$ with $f_0 = f$ and tangent $\dot f_t|_{t=0} = h$. For covariate-direction tangents $h_i = f s_i$,

$$(G_f)_{ii} = \left.\frac{d^2}{dt^2}\right|_{t=0} D_{\mathrm{KL}}(f \| f_{i,t})$$

and summing over $i$ yields $H_G(f)$.

A nonparametric Covariate Cramér–Rao Lower Bound (CRLB) is established:

$\mathrm{AsyCov}(\hat \theta) \succeq G_f^{-1},$

providing fundamental variance limits for regular estimators, contingent on geometric alignment (Cheng et al., 25 Dec 2025).

4. Relation to Efficient Fisher Information in Semi-Parametric Models

In a semi-parametric framework with parameter of interest $\theta = T(f) \in \mathbb R^d$ and nuisance parameter $\eta(f)$, the efficient Fisher information $I_{\mathrm{eff}}$ is the covariance of the efficient score—orthogonal projection of the full score onto the complement of the nuisance tangent space. Under a Geometric Alignment Postulate where the efficient score coincides with covariate scores,

$G_f = I_{\mathrm{eff}}(\theta),$

demonstrating that $G_f$ provides the relevant information bound for semi-parametric estimation (Cheng et al., 25 Dec 2025).

5. cFIM in Autoregressive and Covariate-Dependent Models

In autoregressive models for time series or regression models with endogenous covariates, the cFIM (also called the exact conditional Fisher information matrix) captures the information properly accounting for lagged input dependence. For an order-$p$ logistic autoregressive model with or without exogenous covariates, the exact cFIM is

$$I_c(\vartheta | \mathcal I_0) = \sum_{t=p+1}^T \sum_{y_{-t}} w(\phi_t; \vartheta) \, \phi_t \phi_t^\top Q_t(y_{t-1}, ..., y_{t-p}),$$

with explicit definitions for $w$, $\phi_t$, and the lag-block transition kernels $Q_t$. Efficient recursive computation is possible and essential for finite sample inference, yielding variance estimates and confidence intervals that can be narrower and more accurate than empirical approximation, especially when endogenous structure is non-negligible (Gao et al., 2017).

6. cFIM for Data with Measurement Error in Covariates

In regression and forecasting when both coordinates $(X, Y)$ are measured with error, the cFIM formalism marginalizes over latent variables and propagates all error covariances. The marginal likelihood is characterized by an effective covariance

$$R = C_{YY} - C_{XY}^T T^T - T C_{XY} + T C_{XX} T^T$$

where $T$ is the Jacobian of the model with respect to $x$, and the Fisher information for model parameters is expressed solely in terms of $R$, extending the Fisher approach beyond regimes where covariate errors can be ignored (Heavens et al., 2014).

7. Intrinsic Dimensionality, the Manifold Hypothesis, and Information Capture

The cFIM framework facilitates rigorous investigation of the Manifold Hypothesis (MH), which posits that data in high-dimensional ambient spaces are supported near a lower-dimensional submanifold. The information-capture subspace is characterized by the rank of $G_f$: rank-deficiency in $G_f$ is a testable condition for the MH.

The Information-Capture Ratio—the ratio of the trace of $G_f$ attributable to the effective signal coordinates—serves as a quantitative intrinsic dimensionality estimator, directly enabling the transition from heuristic to testable geometric claims about the structure of high-dimensional data (Cheng et al., 25 Dec 2025).

---

Summary Table: Principal cFIM Formulations

Context	cFIM Formula	Reference
Information geometry	$$(G_f)_{ij} = \mathbb{E}_f[\partial_i\ln f\, \partial_j\ln f]$$	(Cheng et al., 25 Dec 2025)
Logistic autoregression, endogenous input	$$I_c(\vartheta|\mathcal{I}_0) = \sum w(\phi_t;\vartheta)\phi_t\phi_t^\top Q_t$$	(Gao et al., 2017)
Regression with measurement errors	$$F_{\alpha\beta} = \frac{1}{2}\mathrm{Tr}[R^{-1}R_{,\alpha}R^{-1}R_{,\beta}] + \mu_{,\alpha}^T R^{-1} \mu_{,\beta}$$	(Heavens et al., 2014)

The Covariate Fisher Information Matrix thus provides a unifying and operationally tractable platform for modern inference and explainability in scenarios where the role, quality, and structure of the covariates are central to the statistical modeling problem.