Optimal Linear Estimation of Functionals

• The paper presents a novel constrained MLE with principal component truncation to achieve minimax optimal rates in estimating the infinite-dimensional slope function.
• The methodology employs orthonormal basis expansion and careful selection of truncation parameters to balance approximation error and variance.
• Results further validate the use of post-PCA regression and extend its applications to non-Gaussian models, offering practical insights for functional data analysis.

Optimal linear estimation of functionals refers to the statistical theory and methodology for estimating unknown aspects of infinite-dimensional objects—such as regression coefficient functions, or more generally linear or nonlinear functionals—when the data-generating process is modeled by functional regression, nonparametric models, or stochastic processes. In the context of (Dou et al., 2010), the primary focus is on the optimal estimation of an infinite-dimensional slope function B in generalized functional regression models where the response variable's distribution lies in an exponential family, and the canonical parameter is specified as a linear functional of the random functional predictor.

1. Exponential Family Models and Functional Regression

A central modeling framework here is the exponential family with canonical form density $f_\lambda(y) = \exp\{\lambda y - \psi(\lambda)\}$, where $\psi$ is the cumulant generating function. In functional regression for this setting, the canonical parameter for the i-th observation is

$\lambda_i = a + \int_0^1 X_i(t) B(t) dt$

with $X_i$ an observed random function in $L^2[0, 1]$, $B$ an unknown slope function, and $a$ an intercept. This enables the model to encode non-Gaussian responses (e.g., binary, Poisson, binomial) and allows maximal flexibility to incorporate many classical statistical models as special cases, provided they can be cast in the exponential family form.

The mapping from the functional predictor $X_i$ to the canonical parameter $\lambda_i$ through the inner product with $B$ is critical for both modeling and estimation, transforming the statistical task to estimating a linear functional of an infinite-dimensional object.

2. Linear Functional Structure and Dimension Reduction

The estimation of $B$ is technically challenging due to its infinite-dimensional nature. The standard approach exploits the linear structure: $\lambda_i = a + \langle X_i, B \rangle_{L^2[0,1]}$ Both $X_i$ and $B$ are expanded in an orthonormal basis $\{\phi_k\}$, typically the eigenfunctions of the covariance operator $\mathcal{K}$ of $X$, yielding

$$X_i(t) = \sum_{k=1}^\infty x_{ik} \phi_k(t), \quad B(t) = \sum_{k=1}^\infty b_k \phi_k(t)$$

so that the inner product reduces to an infinite sum $\sum_{k} x_{ik} b_k$. This observation naturally leads to truncation: for fixed $N$, take only the span of the first $N$ eigenfunctions. This "post-principal components" reduction step produces a finite-dimensional surrogate problem, yielding an estimator for $B$ as a finite vector—and thus a system amenable to standard, if high-dimensional, parametric estimation strategies.

3. Minimax Optimality and Rates of Convergence

The core theoretical achievement is the derivation of minimax upper and lower bounds for the integrated squared error $\mathbb{E} \| \hat B - B \|_{L^2}^2$ under regularity assumptions characterized by the decay rates of the eigenvalues $\theta_k$ of the covariance operator $\mathcal{K}$ ($\theta_k \asymp k^{-\alpha}$), and of the coefficients $b_k$ in the expansion of $B$ ($b_k \asymp k^{-\beta}$). The minimax risk is of the order

$n^{(1-2\beta)/(\alpha + 2\beta)}$

when $2\beta > 1$. The degree of ill-posedness is governed by $\alpha$, and the smoothness of $B$ is controlled by $\beta$. If the eigenvalue decay is rapid (large $\alpha$), the estimation problem becomes more ill-posed; conversely, a smoother $B$ (large $\beta$) allows for faster rates.

The main theorems show that these rates are attained (up to multiplicative constants) by their constructed estimator and that no estimator can fundamentally outperform this rate within the described model class and regularity regime.

4. Construction of the Estimator

The estimator achieving the optimal rates is constructed via a two-stage truncated and constrained maximum likelihood estimation (MLE):

1. Principal Components Truncation: Project each observed $X_i$ onto the first $N$ eigenfunctions, replacing the infinite-dimensional regression problem with an $N$-dimensional problem. Here, $N$ is taken as a small power of $n$, growing with sample size to ensure the approximation error tends to zero as $n \rightarrow \infty$.
2. Constrained MLE for Slope Coefficients: Estimate the coefficients of $B$ in the truncated basis by maximizing the likelihood for the exponential family model, but over the first $N$ coefficients only. Then, a second truncation is performed: the estimator for $B$ is defined as the expansion using only the first $m$ coefficients of the MLE, where $m$ is optimally calibrated (typically $m \asymp n^{1/(\alpha+2\beta)}$).

The dimension $N$ controls the approximation error (bias) due to truncation, while $m$ controls the variance stemming from parameter estimation in a finite sample. Their interplay yields the minimax rate.

5. Change-of-Measure and Bias Correction

A substantial technical advance is needed due to the nonlinearity induced by the cumulant function $\psi(\cdot)$. Unlike classical linear models, the MLE in the exponential family has an inherent bias—not generally a linear function of the observations due to the nonlinear link. The paper resolves this by invoking a change-of-measure argument, inspired by Le Cam’s asymptotic equivalence theory. By carefully bounding the Hellinger distance between the true model and its truncated approximation, it is established that the loss incurred by working with the finite-dimensional likelihood is asymptotically negligible compared to the estimation error.

This approach justifies analyzing the estimator under the simplified, truncation-induced local model and ensures that the bias inherent to the nonlinear link has no effect on the leading order minimax rate.

6. Practical and Methodological Implications

These theoretical results have deep ramifications for applied functional data analysis:

• Justification for Post-PCA Regression: The widespread practice of principal components regression for functional predictors (i.e., projecting onto leading empirical eigenfunctions before running regression) is theoretically vindicated beyond Gaussian models, encompassing binary and count data.
• Parameter Tuning: The analysis dictates how to select the number of principal components for projection ($N$) and the cut-off $m$ in forming the final estimator of $B$, based on data and plausible smoothness/ill-posedness regimes.
• Non-Gaussian Responses: The theory enables estimation and inference in models such as functional logistic regression, functional Poisson regression, and others, where the response is linked nonlinearly to a functional predictor.
• Broad Applicability: Applications include macroeconomic forecasting (e.g., using yield curves to predict recessions), biomedical settings (e.g., signal-data predictors with binary responses), and any area with high- or infinite-dimensional functional predictors and exponential family-type responses.

The methods rigorously address both computational tractability and statistical optimality. The justified use of finite-dimensional approximations and explicit regularization provides a flexible, efficient, and theoretically sound framework for optimal linear estimation of functionals in modern, structurally rich regression environments.

---

In summary, the results of (Dou et al., 2010) establish that, in exponential family functional regression, judicious application of dimension reduction and constrained MLE—coupled with advanced probabilistic arguments for controlling approximation-induced biases—yields theoretically optimal and practically implementable estimators for infinite-dimensional slope functions, fully quantifying the roles of smoothness and ill-posedness in dictating minimax rates.