Relaxed Mean Squared Error

• Relaxed Mean Squared Error is a risk-aware extension of the classical MMSE that introduces a constraint on the error's fourth moment to control volatility.
• It employs a Lagrangian framework to trade off the mean squared error with higher-order risks, leading to a nonlinear estimator adjusted for skewness and heavy tails.
• Practical illustrations show that RMSE reduces the fourth-moment risk by 25-30% while incurring a modest increase in the mean squared error, enhancing robustness under unstable conditions.

The relaxed mean squared error (RMSE), also termed risk-aware mean squared error, generalizes the standard minimum mean squared error (MMSE) criterion by explicitly introducing a constraint on the volatility of the squared error. Unlike traditional MMSE, which solely minimizes the expected squared deviation between prediction and target, RMSE penalizes the higher-order moments of the error, thereby controlling not only the average performance but also the risk associated with large deviations. This approach is particularly valuable in scenarios where error distributions are skewed or heavy-tailed, and the conventional MMSE estimator lacks stability due to unconstrained error variance or higher moments (Kalogerias et al., 2019).

1. Standard MMSE Estimator

The MMSE estimator seeks a measurable function $f: \mathbb{R} \to \mathbb{R}$ that minimizes the objective

$\mathbb{E}[(X - f(Y))^2]$

where $X$ is the ground truth and $Y$ is the observed variable. By the orthogonality principle, the pointwise optimal estimate is

$f_{\mathrm{MMSE}}(y) = \mathbb{E}[X \mid Y = y].$

The MMSE estimator is risk-neutral in the sense that it optimizes only the mean squared error, and does not account for higher-order moments, such as the variance of the squared error (Kalogerias et al., 2019).

2. Relaxed (Risk-Aware) MSE Criterion

To address the lack of stability inherent in the MMSE estimator under high error volatility, the RMSE criterion introduces a constraint or penalty on the conditional fourth central moment (the variance of the squared error):

$$\mathbb{E}\left[\left( (X-f(Y))^2 - \mathbb{E}[ (X-f(Y))^2 ] \right)^2 \right] = \mathbb{E}[ (X-f(Y))^4 ] - \left(\mathbb{E}[ (X-f(Y))^2 ]\right)^2.$$

For a prescribed risk budget $\rho \ge 0$, the associated optimization is

$$\min_f\ \mathbb{E}\left[ (X-f(Y))^2 \right] \quad \text{subject to}\quad \mathbb{E}\left[ \left((X-f(Y))^2 - \mathbb{E}[(X-f(Y))^2]\right)^2 \right]\le\rho.$$

This leads naturally to a Lagrangian penalized formulation: $$\mathcal{L}(f, \lambda) = \mathbb{E}\left[ (X-f(Y))^2 \right] + \lambda \left( \mathbb{E}[ (X-f(Y))^4 ] - \left( \mathbb{E}[ (X-f(Y))^2 ] \right)^2 - \rho \right),$$ where $\lambda \ge 0$ is the Lagrange multiplier controlling the trade-off between mean performance and risk-averse regularization (Kalogerias et al., 2019).

3. Optimal Estimator under Risk-Aware MSE

Assuming mild moment boundedness, the minimization can be performed pointwise by the interchangeability principle. Defining

$$\mu_1(y) = \mathbb{E}[ X | Y=y ], \qquad \mu_2(y) = \mathbb{E}[ X^2 | Y=y ], \qquad \mu_3(y) = \mathbb{E}[ X^3 | Y=y ],$$

and $\sigma^2(y) = \mu_2(y) - \mu_1(y)^2$, the optimal risk-aware estimator $f^*_\lambda(y)$ solves,

$$(1 + 2 \lambda \sigma^2(y)) \, x = \mu_1(y) + \lambda(\mu_3(y) - \mu_2(y) \mu_1(y)),$$

yielding the explicit expression: $$f^*_\lambda(y) = \frac{ \mu_1(y) + \lambda \big[ \mu_3(y) - \mu_2(y) \mu_1(y) \big] }{ 1 + 2 \lambda \sigma^2(y) }.$$

This estimator is a nonlinear, regularized variant of the standard MMSE: it introduces a bias depending on the third conditional moment, and a shrinkage factor inversely related to the conditional variance. Notably, in terms of $p_{X|Y}(x|y)$,

$$f^*_\lambda(y) = \frac{ \int x p_{X|Y}(x|y) dx + \lambda \left( \int x^3 p_{X|Y}(x|y) dx - \int x^2 p_{X|Y}(x|y) dx \int x p_{X|Y}(x|y) dx \right) }{ 1 + 2\lambda \left( \int x^2 p_{X|Y}(x|y) dx - \big(\int x p_{X|Y}(x|y) dx\big)^2 \right) }.$$

(Kalogerias et al., 2019)

4. Existence, Uniqueness, and Theoretical Guarantees

The explicit construction above is justified under the following moment and regularity conditions:

• $\mathbb{E}[|X|^3] < \infty$ ensures finiteness of all required conditional moments.
• $\mathbb{E}[\mu_3(Y)^2] < \infty$ guarantees square-integrability of the third-moment filter.
• A Slater-type strict feasibility condition ensures that a dual-optimal Lagrange multiplier $\lambda^*$ exists, guaranteeing zero duality gap and uniqueness.

Under these hypotheses, the minimization is strictly convex in $f(y)$ and possesses a unique solution almost surely with respect to $Y$ (Kalogerias et al., 2019).

5. Practical Illustrations and Regime-Specific Behavior

The risk-aware estimator demonstrates distinct advantages over risk-neutral MMSE in models characterized by high skewness or heavy tails:

• Skewed State-Dependent Noise:

For $X\sim\mathrm{Exp}(1/2)$, $Y|X\sim \mathcal{N}(X,9X^2)$, the standard MMSE $f_{\mathrm{MMSE}}(y)$, being symmetric, can yield large estimation errors for small $y$, where rare large $X$ may be observed through significant noise. The risk-aware estimator $f^*_\lambda(y)$ hedges against variance in the tails by over-estimating for small and large $y$. This results in a 25–30% reduction in the conditional fourth-moment risk, at the expense of a 10–20% increase in the mean squared error.

• Heavy-Tailed Priors:

For $X \sim$ Student-$t$ ($\nu < 4$), $Y = X + v$ with $v\sim\mathcal{N}(0,1)$, the posterior $p_{X|Y=y}$ inherits heavy tails. The standard MMSE can be shifted by outlying $y$. The risk-aware estimator shrinks large-magnitude estimates toward zero, sharply reducing the fourth-moment risk with only a moderate increase in mean squared error.

In these examples, the RMSE estimator achieves improved robustness to outlier-induced volatility, supporting its application where error stability is a concern (Kalogerias et al., 2019).

6. Tuning and Trade-off Interpretation

The Lagrange multiplier $\lambda \ge 0$ (or equivalently, the risk budget $\rho$) directly parameterizes the trade-off between average squared error and higher-order predictive risk. As $\lambda$ increases, the impact of the penalty for error volatility becomes more pronounced, biasing the estimator toward risk-averse predictions. $\lambda$ can be set to ensure that the constraint on the fourth-moment risk is satisfied with equality. This formulation enables explicit control over the tail behavior of the estimation error, making RMSE suitable for applications requiring stability against rare but significant deviations (Kalogerias et al., 2019).