Pseudo-Inverse Learning Rule

• Pseudo-Inverse Learning Rule is a closed-form method that computes neural network parameters using the Moore–Penrose pseudo-inverse.
• It achieves rapid least-squares convergence and deterministic outcomes, eliminating iterative gradient descent and step-size tuning.
• Modern adaptations incorporate regularization, online updates, and extensions to deep and quantum architectures for enhanced robustness.

The pseudo-inverse learning rule is a family of algebraic, closed-form training methods for neural networks and related models, wherein parameter updates are computed via the Moore–Penrose pseudo-inverse of an appropriate Jacobian or feature matrix rather than through incremental, iterative optimization. This approach, which encompasses both classical feedforward networks and contemporary architectures (including quantum neural networks), offers a principled alternative to gradient-based optimization, yielding deterministic solutions, rapid convergence to least-squares optima, and elimination of step-size hyperparameters. Modern pseudo-inverse schemes employ regularization for numerical stability and generalization, support deep and auto-encoding architectures, and extend to adaptive, online, and hardware-constrained settings.

1. Algebraic Foundations and General Schema

At its core, the pseudo-inverse learning rule recasts network training as a (linearized or exact) inverse problem. For a model mapping inputs $X \in \mathbb{R}^{N \times d}$ to outputs $Y \in \mathbb{R}^{N \times m}$ via a feature (or hidden) matrix $H$, one seeks to minimize a (possibly regularized) quadratic loss:

$$\underset{W}{\min} \| H W - Y \|^2 + \lambda \|W\|^2.$$

The closed-form solution is

$W = (H^T H + \lambda I)^{-1} H^T Y,$

or, equivalently, $W = H^+ Y$ where $H^+$ is the Moore–Penrose pseudo-inverse. In multilayer or autoencoding contexts, this inversion is performed layerwise, with $H$ constructed recursively via nonlinear activations (Guo, 2018, Guo et al., 2018, Liu et al., 2024).

In quantum neural networks (QNNs), the rule is applied in probability space: the Born rule-derived model outputs $p(x; \theta)$, and parameter corrections $\Delta\theta$ are computed by locally inverting the Jacobian $J = \partial p/\partial\theta$ using the pseudo-inverse of $J$ (Seo, 23 Jan 2026).

2. Classical Feedforward Networks and Variants

Single Hidden Layer Networks and ELM

For single hidden layer feedforward networks (SHLN) or Extreme Learning Machines (ELM), input-to-hidden weights are fixed (often random and normalized), hidden activations $H$ are computed, and output weights $W$ are set by pseudo-inverse: | Step | Description | Formula or Method | |------------------------|---------------------------------------------|--------------------------------------------| | Input-to-hidden | Fixed (random, orthonormal, or $X^+$) | $H = \phi(XV)$ | | Hidden-to-output | Solved by pseudo-inverse | $W = H^+ T$ |

Variants differ in precise initialization and regularization of $V$, and in whether the output nonlinearity is invertible (e.g., requiring application of $\phi^{-1}$ at output) (Guo, 2018, Guo et al., 2018, Cancelliere et al., 2015).

Regularization and Rank

Numerical instability can occur if $H$ is close to singular; Tikhonov regularization ($\lambda > 0$) stabilizes the inversion and improves generalization. The hidden layer width must be chosen judiciously—too large leads to rank deficiency and overfitting, as diagnosed by singular value analysis or monitoring validation error spikes ("critical hidden layer size") (Cancelliere et al., 2015). Scaling of input weights, e.g., $c_{ij} \sim \mathcal{U}(-1/\sqrt{M}, 1/\sqrt{M})$, prevents activation saturation at large hidden widths.

Deep, Autoencoder, and Hybrid Architectures

Pseudo-inverse learning is extended to stacked autoencoders ("PILAE" (Guo et al., 2018)) and deep multilayer networks. Here, the encoder at each layer is a (possibly low-rank) pseudo-inverse of its input; the decoder solves a least-squares minimization:

$$\text{Encoder:}~W_e = \text{low-rank}(X^+), \quad \text{Decoder:}~W_d = X H^+(=\arg\min_{W_d} \|W_d H - X\|^2).$$

Layer depth and hidden size can be chosen adaptively via data-driven heuristics (e.g., SVD-based rank truncation or validation error monitoring) (Guo et al., 2018, Liu et al., 2024).

3. Adaptive, Online, and Inverse-Free Algorithms

Online Pseudoinverse Learning

Online and incremental update versions of the pseudo-inverse rule, such as the OLP (Online Pseudoinverse Learning) or Greville's algorithm, update the output weight $W$ and an inhibition matrix $\Theta$ at each new data point, maintaining the closed-form solution in a streaming context:

• Weight update:

$$W_k = W_{k-1} + (y_k^T - W_{k-1} a_k) (\Theta_{k-1} a_k)^T / (1 + a_k^T \Theta_{k-1} a_k)$$

• Inhibition matrix update:

$$\Theta_k = \Theta_{k-1} - (\Theta_{k-1} a_k)(\Theta_{k-1} a_k)^T / (1 + a_k^T \Theta_{k-1} a_k) + \alpha_k I$$

The algorithm is significantly more memory-efficient than batch SVD computation, supports adaptation to non-stationary data via $\alpha_k$, and exhibits biological plausibility (distributed representation, local error-driven updates) (Tapson et al., 2012).

Inverse-Free Blocks and Factorization

Recursive block-inverse and LDL$^T$ factorization methods incrementally maintain the regularized inverse or pseudo-inverse as new hidden nodes or data arrive, eliminating the need for explicit matrix inversion at each step. These schemes substantially accelerate ELM-like methods, with complexity per addition reduced by factors of $3\times$ or more, and superior numerical stability under long sequences of updates (Zhu et al., 2019).

4. Quantum and Non-classical Models

In QNNs, the pseudo-inverse learning rule is formulated in probability space. For a set of input–target pairs $(x_i, y_i)$, the model outputs Born-rule probabilities $p(x_i; \theta)$. The gradient (Jacobian) $J_{ij} = \partial p(x_i;\theta)/\partial \theta_j$ is computed using the parameter-shift rule (for quantum parameters), and parameter updates are determined as the solution to:

$$\Delta\theta = \arg\min_{\Delta\theta} \|y - p(\theta) - J\Delta\theta\|^2 + \lambda \|\Delta\theta\|^2$$

yielding the regularized pseudo-inverse update:

$$\Delta\theta = (J^T J + \lambda I)^{-1} J^T (y - p(\theta))$$

This covariant update requires no learning rate, achieves rapid reduction of the loss, and is robust to hardware noise and shot-sampling effects (Seo, 23 Jan 2026).

Comparison with gradient-based optimization reveals several key differences:

• Pseudo-inverse moves in the full column-space of $J$ and directly cancels the residual in one linearized step.
• Gradient descent and Adam require many small steps, are sensitive to loss landscape geometry, and need $\eta$ tuning.
• Pseudo-inverse approaches typically achieve low loss within $5$–$10$ updates, whereas GD/Adam require $\gtrsim 100$ (Seo, 23 Jan 2026).

5. Extensions, Limitations, and Special Cases

Hopfield-Type Networks and Memory/Generalization Tradeoff

In attractor and Hopfield-style networks, the pseudo-inverse rule is used both for fixed-point and cyclic-attractor storage. For a set of binary patterns $\Sigma$, the synaptic matrix is constructed via $J = \Sigma \Sigma^+$. Modifications include replacing empirical pattern covariances with expected correlations for families of noisy examples, enabling robust generalization to archetypal "concepts" instead of mere memorization. The resulting dynamics exhibit phases corresponding to pure retrieval and several forms of concept generalization (fully symmetric, class-representant, outlier-excluding), with strong replica-symmetry-breaking effects and a nuanced tradeoff between memory capacity and robustness to noise (Zhang et al., 2013, Benedetti et al., 1 Feb 2026).

Deep Non-gradient Architectures

Multiway architectures (e.g., semi-adaptive synergetic two-way systems) combine forward closed-form encoding, backward label-propagation (via generalized inverses), and feature fusion modules, further enhancing representational capacity and automating architecture/design selection. Each module weights are solved by generalized inverse, regularized per-layer, and subsystems can be grown or pruned adaptively. These ensembles may achieve competitive or superior empirical accuracy relative to classical deep learning models while reducing tuning and significantly shortening training times (Liu et al., 2024).

Analytical and Practical Limitations

• Computational cost scales cubically with the number of features or parameters for full pseudo-inverse updates (batch case).
• For very large networks or datasets, low-rank or randomized/sketching approaches, blockwise updates, or iterative refinements are advantageous.
• In deep or highly nonlinear settings, the linearization underlying the pseudo-inverse step is only locally accurate, and large residuals may degrade performance.
• Care is required to avoid overfitting: regularization ($\lambda$), validation-derived stopping, and rank/width control remain necessary.

6. Comparative Experimental Insights

Extensive comparative studies demonstrate that pseudo-inverse-based approaches converge to global minima on mean-squared or cross-entropy loss, often within an order of magnitude fewer iterations than gradient methods. For example, in both shallow and deep tasks, algebraic/closed-form learning achieves classification errors within $1$–$2\%$ of best-tuned iterative schemes, but with up to $10$–$100\times$ faster training times (Guo et al., 2018, Liu et al., 2024). On quantum benchmarks, the pseudo-inverse rule scales its final mean-squared error as $1/S$ with shots $S$, whereas Adam stagnates well above optimality under finite-sample or noise constraints (Seo, 23 Jan 2026). Adaptive or online pseudo-inverse rules permit efficient streaming learning, rapid adaptation to nonstationary environments, and memory-efficient deployment (Tapson et al., 2012, Zhu et al., 2019).

7. Broader Implications and Theoretical Perspective

The pseudo-inverse learning rule offers a unified, mathematically tractable alternative to procedural optimization for a broad spectrum of machine learning models—linear, nonlinear, quantum, deep, and adaptive. By exploiting analytic least-squares solutions, it circumvents several of the canonical weaknesses of gradient descent, including slow convergence, sensitivity to hyperparameters, vanishing/exploding gradients, and entrapment in local minima. It is particularly effective in scenarios where computational resources or hyperparameter tuning budgets are constraining, or where strict determinism and analytic solvability are required. Current research aims at scaling such methods to even deeper and broader models, hybridizing with gradient-based updates for increased flexibility, and fully exploiting their analytic properties for robust architecture and feature-learning automation (Guo, 2018, Liu et al., 2024, Seo, 23 Jan 2026).