Zero-Initialized Regressor Overview

• Zero-initialized regressor is a regression model with all parameters starting at zero, offering an unbiased baseline particularly critical in low-rank adaptations.
• It is widely applied in matrix factorization, factor analysis, and LoRA-based models to ensure stable, neutral starting points in iterative optimization.
• Empirical results and EM algorithm analyses demonstrate that zero initialization promotes reliable convergence and mitigates risks of early divergence.

A zero-initialized regressor is a regression model or module whose parameters—such as weights in a neural network, coefficients of a linear model, or factors in a low-rank representation—are initialized to zero at the start of training or estimation. This initialization strategy is especially pertinent in low-rank adaptation, matrix/tensor completion, and multivariate statistical modeling where constraints or regularization render parameter initialization a key factor in empirical behavior and optimization. The following sections detail the formal context, mathematical properties, algorithmic implications, and empirical considerations for zero-initialized regressors in representative low-rank settings.

1. Formal Definition and Contexts

A zero-initialized regressor is defined by the initial condition that—for every learnable parameter θ within the regression mapping $f_\theta(\cdot)$—$\theta_0 = 0$. In neural low-rank adaptation, such as LoRA (Low-Rank Adaptation), this manifests as initializing the decomposition matrices $A$ and $B$ such that, typically, $B=0$ and $A$ is either zero or set according to a narrow Gaussian (Shao et al., 6 Aug 2025). In classic matrix factorization or factor analysis, zero-initialization refers to loading matrices and/or specific sub-blocks initialized as the zero matrix prior to EM or gradient-based optimization (Ahfock et al., 2021).

Zero-initialization is relevant in settings where symmetry or invariance is preserved, or where the regression mapping is affine and the statistical aggregate (such as covariance) is central to estimation. In many low-rank algorithms, the initial zero-filled state provides a stable reference point for subsequent adaptation or EM iteration.

2. Mathematical Properties and Structural Implications

The mathematical effects of zero-initialization depend on the geometry of the regression problem and the properties of the update scheme:

• Symmetry and Identifiability: In factor analysis with Gaussian noise and low-rank structure as in (Ahfock et al., 2021), initialization with $\Lambda=0$ or proper structured zeros for loading matrices does not bias the EM algorithm's asymptotic convergence, since likelihood surfaces in low-rank Gaussian settings often have symmetric basins under orthogonal transformations of latent factors.
• Lower-boundedness: In EM-based factor analysis, initial zeros for the unique variances $\Psi$ or loadings $\Lambda$ correspond to an initial model with identity or diagonal covariance, which is a proper statistical model and provides a lower-bound starting point for the log-likelihood maximization (Ahfock et al., 2021).
• Parameter Recovery: In LoRA and matrix/tensor completion, zero initialization of adaptation matrices (e.g., $B=0$ in $ΔW=BA$) ensures that the initial adapted weights $W' = W$ coincide with the pretrained backbone, avoiding early divergence or catastrophic forgetting (Shao et al., 6 Aug 2025).

A plausible implication is that in highly non-convex or multimodal objective landscapes, zero initialization may avoid certain pathological local minima, but may also slow initial convergence if the model class is expressive and the optimum is not proximate to zero.

3. Algorithmic Considerations

Zero-initialized regressors are encountered in Expectation-Maximization (EM), meta-learning, and end-to-end optimization of low-rank decompositions:

• EM Algorithm in Factor Analysis: In the statistical file-matching setting (Ahfock et al., 2021), initial values for $\Lambda$ and $\Psi$ are required for iterative EM. Zero initialization for $\Lambda$ (followed by suitable positive entries for $\Psi$) provides a neutral starting point, and the EM updates (using completed scatter matrices and Gaussian-factored conditional updates) ensure monotonic likelihood improvement.
• LoRA Adaptation in Transformers: In LoRA-based adaptation, a conventional regimen is to initialize $B=0$ and $A$ from $N(0,σ^2)$ (often with mean zero, possibly variance close to zero), such that the influence of LoRA is initially canceled and only becomes nonzero as $A,B$ are trained (Shao et al., 6 Aug 2025). This guarantees that the downstream model starts as a strict copy of the backbone.
• Low-Rank Multimodal Fusion: In low-rank multimodal fusion (Liu et al., 2018), weight factor matrices $W_m^{(i)}$ are initialized (typically Xavier or random) for forward/backward propagation. The strategy of near-zero initialization, though not strictly zero, shares the principle of unbiasedness and mitigates over-influencing outputs prior to learning.

Algorithmically, zero initialization is trivial to implement and computationally light, but requires careful monitoring for stagnation, especially in models with ReLU or similar nonlinearities with zero-gradient plateaus.

4. Empirical Performance and Role in Benchmark Results

When benchmarking low-rank fusion and matrix-completion algorithms, the initialization scheme critically affects both convergence speed and empirical error. Key empirical findings include:

• In factor analysis for file-matching (Ahfock et al., 2021), good initialization (e.g., from complete-case factor analysis or small randomization around zero) is recommended to avoid poor local modes, but the model achieves tightly clustered low mean-squared error even when initialized conservatively. For example, in the Sachs dataset, factor analysis median error is $6\times 10^{-4}$, substantially lower than other methods.
• In LoRA fusion and multi-task adaptation (Shao et al., 6 Aug 2025), LoRA’s default use of $B=0$ ensures unchanged performance at the beginning of adaptation, and experimental results show that models are robust to this convention. ICM-Fusion’s meta-fusion procedure begins from that baseline and does not detract from the original performance with zero initialization.

A plausible implication is that, for low-rank models under strong regularization or identifiable signal, initialization at or near zero does not impede model recovery and may in fact improve robustness to overfitting or label leakage.

5. Connections to Low-Rank Structure and Model Fusion

Zero-initialized regressors are most relevant in the context of models explicitly or implicitly defined as low-rank—either via latent variable structures (as in factor analysis), matrix factorization (as in LoRA), or tensor decomposition (as in multimodal fusion). In these frameworks:

• Zero-initialized low-rank factors allow the model to begin as uninformative or equivalent to an unadapted base, ensuring that all predictive signal must be learned from data.
• In fusion settings, especially in file-matching or domain adaptation where new tasks or modalities are continually incorporated, zero initialization imposes no shift on the base model’s input-output mapping before observed data for the new domain or task.

Table 1 summarizes roles and effects across representative algorithms:

Model Class	Param Initialization	Empirical Effect
Factor Analysis (EM)	$\Lambda=0$, $\Psi>0$	Neutral; monotonic EM progress
LoRA Adaptation	$B=0$, $A$ near-zero	Backbone unchanged at start; stable fusion
Low-Rank Multimodal Fusion	$W_m^{(i)}$ ≈ 0	Unbiased early outputs; quick stabilization

6. Theoretical and Practical Implications

The use of zero-initialized regressors is justified under:

• Theoretical guarantees: EM and backpropagation with such initialization do not restrict the attainable optimum, provided learning dynamics can escape strict saddle points and the data are sufficiently informative (Ahfock et al., 2021).
• Practical stability: In low-rank settings, initial zero models avoid spurious prediction before any learning occurs. This is especially important in incremental, meta-learning, or transfer pipelines, as implemented in domain fusion and adapter merging (Shao et al., 6 Aug 2025).

A plausible implication is that, while not mandatory for optimality, zero-initialization can be regarded as a default, stable choice in applied low-rank regression when strong priors or batch normalization are not present.

7. Related Concepts and Alternative Strategies

While zero initialization is a standard baseline, alternative strategies include random small-variance initialization (e.g., Xavier or He schemes for deep models) and pre-training-based initialization (using parameters fit on complete cases or warm-started from auxiliary data) (Ahfock et al., 2021). The selection can influence convergence behavior and early-stage performance, but in low-rank or latent variable contexts, zero-initialization remains a widely adopted, theoretically valid, and empirically robust default for regressors in multivariate fusion and adaptation frameworks.