RBLQNN: ReLU-Bias Loss Quantile Neural Network

• The paper demonstrates that RBLQNN significantly reduces quantile MAE and CRPS across complex datasets by employing quantile counter-balancing and ReLU bias loss.
• The method uses a standard MLP architecture augmented with specialized loss function modifications for precise non-Gaussian uncertainty estimation in climate and geophysical contexts.
• Empirical results show a 10–20% CRPS reduction and minimized quantile crossing, outperforming baseline models like LQR and MVE.

A ReLU-Bias Loss Quantile Neural Network (RBLQNN) is a quantile regression neural network framework designed for flexible and accurate estimation of predictive uncertainty. Its architecture is a standard multilayer perceptron (MLP) with two key modifications to the loss function: quantile counter-balancing and a "ReLU bias" penalty to enforce monotonicity among predicted quantiles. RBLQNN is developed to address limitations of conventional quantile regression and mean-variance neural network methods in capturing nonlinear and non-Gaussian conditional distributions, with particular success demonstrated for climate and geophysical data (Brettin et al., 24 Jan 2026).

1. Model Architecture

The RBLQNN utilizes a classical feed-forward MLP that maps a $p$-dimensional input $x$ to $m$ output nodes, each providing an estimate at a specified quantile level $q_j$. The functional form is:

$$f_\theta : \mathbb{R}^p \to \mathbb{R}^m, \quad f_\theta(x) = (\hat{y}_{q_1}, \ldots, \hat{y}_{q_m}),$$

where $q_j \in \{ q_1,\ldots, q_m \}$ are predetermined quantile levels.

Hidden layers employ ReLU activations, $ReLU(z) = \max\{0, z\}$, and only the final layer is linear. Special architectural features such as monotonic output layers are not required; avoidance of quantile crossing is delegated to the loss function. The standard implementation predicts $m=19$ evenly spaced quantiles $q_j = 0.05, 0.10, ..., 0.95$\; extrapolated quantiles $q=0$ and $q=1$ are associated with $-\infty$ and $+\infty$, respectively.

2. Loss Function Design

The RBLQNN loss function extends standard quantile loss (the "pinball" loss) with two innovations:

• Quantile Counter-Balancing: To correct for differential weighting of extremal quantiles in standard pinball loss, each term is rescaled by a quantile-specific factor $\lambda_j$, approximated assuming normality:

$\lambda_j = \exp(\Phi^{-1}(q_j)/2),$

where $\Phi^{-1}$ is the standard normal quantile function.

• ReLU Bias Loss: To penalize non-monotonic quantile outputs (crossings), an additional term penalizes violations where $\hat{y}_{q_j} > \hat{y}_{q_{j+1}}$:

$$\mathcal{L}_{ReLU}(\hat{y}_q) = \sum_{j=1}^{m-1} \max\{0, \hat{y}_{q_j} - \hat{y}_{q_{j+1}}\}.$$

The total objective per sample is:

$$\mathcal{L}(y, \hat{y}_q) = \mathcal{L}_Q(y, \hat{y}_q) + \eta \mathcal{L}_{ReLU}(\hat{y}_q)$$

where $\mathcal{L}_Q$ is the composite quantile pinball loss, and $\eta>0$ is a tunable hyperparameter.

These modifications are designed to ensure uniform accuracy across quantiles and to minimize degenerate or non-physical predictive distributions. Unlike strictly "hard" approaches to monotonicity, the ReLU bias loss is a soft penalty, trading rare crossings for improved optimization transparency and implementation simplicity.

3. Training Regimen and Implementation

Training uses the Adam optimizer. Initialization includes a brief warm-up period where the loss is temporarily switched to mean squared error (MSE) to stabilize the initial stages. Model selection is conducted through early stopping, preserving weights with the lowest validation loss.

Hyperparameters include learning rates ($10^{-8}$ to $10^{-1}$), batch sizes ($128$–$512$), up to $200$ epochs with early-stop patience, and optional regularization (weight decay or dropout). All continuous inputs and target variables are standardized to zero mean and unit variance on a per-dataset or per-station basis.

Key practical considerations include the robustness of $\lambda_j$ weights to deviations from normality and the elimination of the need for specialized monotonic output constraints, simplifying implementation.

4. Evaluation Metrics and Comparative Baselines

Model evaluation is performed through several established quantile regression metrics:

• Quantile MAE: Mean absolute error between predicted and empirical quantiles, $$MAE_q = (1/n) \sum_{i=1}^n |y_{q,i} - \hat{y}_{q,i}|$$.
• Calibration via PIT histograms: The observed proportions $r_k$ of samples in bins formed by predicted quantiles are compared against uniformity using $D = \sqrt{(1/B) \sum_{k=0}^{m}(r_k - 1/B)^2}$.
• CRPS: The continuous ranked probability score for quantile outputs, as in Laio & Tamea (2007):

$$CRPS(\hat{y}_q, y) = \frac{2}{m} \sum_{j=1}^m \big[ q_j(y - \hat{y}_{q_j}) 1_{y>\hat{y}_{q_j}} + (1-q_j)(\hat{y}_{q_j} - y) 1_{y \leq \hat{y}_{q_j}} \big].$$

Baselines for comparison include Linear Quantile Regression (LQR), Mean-Variance Estimation (MVE) neural networks, and deterministic (MSE) regression. LQR is fit via composite quantile regression; MVE optimizes a Gaussian log-likelihood, transforming parameter outputs to quantiles via the normal CDF inverse.

The table summarizes the distinguishing characteristics:

Model	Captures Nonlinearity	Captures Non-Gaussianity	Penalizes Crossings
LQR	No	No	No
MVE	Yes	No	N/A
RBLQNN	Yes	Yes	Yes

5. Empirical Findings

The RBLQNN’s efficacy is demonstrated using synthetic and observational geophysical datasets:

• Synthetic Benchmarks: On problems with strong nonlinearity and/or non-Gaussianity (Gumbel noise, Beta-distributed outputs, bimodal Boltzmann system), RBLQNN attains uniformly lower quantile MAE than LQR or MVE, accurately capturing nonlinear and multimodal dependencies. Variants lacking either quantile counter-balancing or ReLU bias loss show marked degradation in quantile crossing rate and prediction error (<3% crossing in standard RBLQNN).
• Daily Maximum Temperature (GSOD): Applied to 1,501 NOAA GSOD weather station datasets with covariates including SLP and geopotential heights, RBLQNN achieves a 10–20% CRPS reduction over climatology and MSE at 99% of stations. Compared to LQR, CRPS is lower at 95% of stations (significantly so at 63%), and comparable to MVE nets at the majority of stations—reflecting that most temperature uncertainties are near-Gaussian except in regimes with nontrivial skewness or kurtosis.
• Precipitation (TRMM): On regression of precipitation-related variables (including ERA5 humidity, temperature, and CAPE) for 220,000 test points, RBLQNN achieves a CRPS of $0.43 \times$ climatology, outperforming LQR ($1.07 \times$), MVE ($1.12 \times$), and MSE ($0.83 \times$). Bootstrap resampling confirms superiority in all pairwise comparisons.

Failure modes are observed in highly deterministic or highly stochastic regimes, where Gaussian MVE may slightly outperform due to the near-Gaussianity or near-triviality of the predictive distribution.

6. Limitations and Prospective Directions

While the RBLQNN incorporates a soft penalty for quantile crossing, rare crossings may still occur. Strictly enforcing monotonicity via architectural modifications, such as cumulative-increment schemes, could further limit crossings at the expense of simplicity.

Metrics such as CRPS require large sample sizes for sensitivity to moderate improvements; development of efficient proper scoring rules for finite-data regimes remains an open challenge. Extending RBLQNN to spatial–temporal architectures (e.g., CNN or RNN variants), Bayesian quantile neural networks, or incorporating physical priors and monotonic spline layers indicate promising research avenues.

7. Significance in Uncertainty Estimation

The RBLQNN presents a general approach to quantile regression with demonstrated flexibility, training stability, and predictive power in scenarios with nonlinear and non-Gaussian uncertainty structures. It obviates the need for architectural monotonic constraints and offers practical advantages for geophysical and climate uncertainty estimation tasks, including operational settings where robustness across a wide hyperparameter range is essential (Brettin et al., 24 Jan 2026). The method's empirical performance highlights its utility for problems where analytic forms are unknown or baseline mean-variance or linear approaches are inadequate.