Surjective Pseudo-Invertible Neural Networks

• SPNNs are neural architectures that extend the Moore–Penrose pseudoinverse to nonlinear settings using explicit pseudo-inverse constructions.
• They utilize surjective layers and bijective completions to achieve zero-shot inversion, semantic control, and robust posterior inference.
• Empirical results show SPNNs excel in multimodal, physical, and generative tasks, overcoming limitations of naive inversion methods.

Surjective Pseudo-Invertible Neural Networks (SPNNs) are a principled class of neural architectures that guarantee surjectivity—every target output is reachable—and admit explicit, tractable pseudo-inverses that generalize the Moore–Penrose pseudoinverse to nonlinear and high-dimensional settings. SPNNs combine rigorous function-theoretic guarantees (surjectivity, consistency, null-space control) with architectural and algorithmic machinery (bijective completions, non-linear back-projection) to enable zero-shot inversion, posterior inference, and semantic-level control in both regression and generative tasks. SPNNs unify perspectives from operator theory, generative modeling, and inverse problem literature under a geometric framework for non-linear invertibility and back-projection.

1. Theoretical Foundations: Surjectivity and Pseudo-Inverses

SPNNs arise from the need to extend the Moore–Penrose pseudoinverse, $A^+$, which solves $Ax = y$ by null-space back-projection, to nonlinear mappings $f: X \to Y$ typical in deep networks. In the linear regime, $x' = x + A^+(y - A x)$ orthogonally projects any $x$ to the closest solution consistent with $y$.

The nonlinear generalization targets maps $g : X \to Y$ that are surjective—every $y \in Y$ admits some $x$ with $g(x) = y$—and defines a "natural" pseudo-inverse $Ax = y$0 based on a bijective completion $Ax = y$1, where $Ax = y$2 parameterizes the null-space. The natural pseudo-inverse is defined by minimizing distance in the $Ax = y$3-representation:

$Ax = y$4

selecting for each $Ax = y$5 the pre-image $Ax = y$6 whose joint state $Ax = y$7 (with $Ax = y$8 the null-space coordinate extractor) is closest to the origin. This construction recovers $Ax = y$9 when $f: X \to Y$0 is linear and is characterized by two reflexive identities:

• $f: X \to Y$1 for all $f: X \to Y$2 (right-inverse),
• $f: X \to Y$3 for $f: X \to Y$4 in the chosen section (left-inverse) (Ehrlich et al., 5 Feb 2026).

2. Surjectivity and Invertibility Conditions in Neural Networks

Surjectivity, the guarantee that every output is attainable, depends on both network architecture and parameter choice. For feedforward and attention-based architectures:

• Pre-LayerNorm residual blocks: For any $f: X \to Y$5, $f: X \to Y$6 is surjective for all continuous $f: X \to Y$7; this follows from Brouwer’s fixed-point theorem, since $f: X \to Y$8 ensures boundedness and $f: X \to Y$9 maps large balls into themselves (Jiang et al., 26 Aug 2025).
• Linear-attention/RetNet blocks: Maps of the form $x' = x + A^+(y - A x)$0 with $x' = x + A^+(y - A x)$1 are almost always surjective for generic full-rank parameters, by degree-theoretic arguments (Jiang et al., 26 Aug 2025).
• Operator-theoretic layers: Sufficient conditions for surjectivity and invertibility involve the use of pointwise bijective activations (e.g., LeakyReLU), Fredholm operators of index zero for infinite-dimensional maps, and contraction/coercivity arguments (via Banach or Leray–Schauder fixed-point theorems). Explicit layerwise surjectivity is constructed via networks $x' = x + A^+(y - A x)$2, with $x' = x + A^+(y - A x)$3 injective and suitable output-input rank ratios maintained for finite-rank truncations (Furuya et al., 2023).

These results imply that most standard deep learning architectures (e.g., Transformer blocks, diffusion model ODE steps) are surjective almost everywhere in parameter space and can in principle be pseudo-inverted (Jiang et al., 26 Aug 2025).

3. SPNN Architecture and Training Procedures

An SPNN consists of surjective blocks, each with explicit pseudo-inverse structure. A canonical SPNN block (in $x' = x + A^+(y - A x)$4 with $x' = x + A^+(y - A x)$5) operates as:

• Forward ($x' = x + A^+(y - A x)$6): $x' = x + A^+(y - A x)$7, where $x' = x + A^+(y - A x)$8, $x' = x + A^+(y - A x)$9 are (arbitrary) neural nets.
• Inverse ($x$0): For given $x$1, estimate null coordinates $x$2 via auxiliary net $x$3, and $x$4, then concatenate $x$5 (Ehrlich et al., 5 Feb 2026).

The training proceeds in two phases:

1. Phase I ("forward"): Train $x$6 (and any mixing transforms) for the primary task (e.g., supervised task loss).
2. Phase II ("inverse"): Freeze $x$7 and optimize $x$8 to enforce "naturality"—aligning $x$9 near $y$0 (the canonical origin in $y$1)—plus auxiliary losses for surjectivity consistency ($y$2) and stability ($y$3).

Architectural enhancements include multi-scale processing (e.g., pixel-unshuffle), orthogonal mixing (tunable basis for signal vs. null-space), and block stacking for higher expressivity. For infinite-dimensional operator settings, each operator is implemented via a finite-rank basis expansion, with injectivity/surjectivity preserved under the appropriate rank and contraction/coercivity conditions (Furuya et al., 2023).

4. Non-Linear Back-Projection and Null-Space Manipulation

SPNNs formalize non-linear back-projection (NLBP), extending the classic linear update $y$4 to nonlinear, surjective maps. Given a bijective completion $y$5, the non-linear back-projection operator is:

$y$6

which updates $y$7 to the unique solution with $y$8 and $y$9 closest to $g : X \to Y$0 in the natural metric. Concretely, the update changes only the $g : X \to Y$1 coordinate in $g : X \to Y$2-space (signal), leaving the null-space untouched, and projects orthogonally onto the solution manifold (Ehrlich et al., 5 Feb 2026).

This operator underpins zero-shot inversion algorithms, where NLBP is incorporated stepwise in DDPM diffusion models to enforce semantic constraints or restore outputs from severely degraded observations (e.g., inversion of classifiers, attribute-guided image editing).

5. Empirical Results and Application Domains

SPNNs have been tested on a variety of inverse problems and generative tasks:

• Synthetic multimodal posteriors: SPNNs exactly recover all modes in ambiguous inverse problems (e.g., 8-mode Gaussian mixtures mapped to 4 labels), without mode collapse (Ardizzone et al., 2018).
• Physical system inference: In 2D inverse kinematics of a four-joint arm, SPNNs correctly recover both "elbow-up/down" solutions, outperforming cVAE and ABC in re-simulation and calibration error (Ardizzone et al., 2018).
• Tissue parameter estimation: For the forward map (tissue param $g : X \to Y$3 multispectral measurement), SPNNs learn $g : X \to Y$4, identifying unrecoverable parameters and nonlinear correlations (e.g., $g : X \to Y$5 and $g : X \to Y$6 trade-off), with best-in-class MAP-RMSE and calibration error (Ardizzone et al., 2018).
• Astrophysical simulations: High-dimensional measurement-to-parameter inference in star formation feedback recover multimodal and highly correlated posteriors (Ardizzone et al., 2018).
• Semantic image restoration: SPNNs trained to invert classifier logits back to faces (with DDPM + NLBP) achieve >92% attribute agreement and plausible reconstructions (Ehrlich et al., 5 Feb 2026).
• Attribute-controlled generation: Semantic editing in generative models by manipulating target $g : X \to Y$7 and projecting onto the solution set via NLBP produces diverse, constraint-satisfying outputs (Ehrlich et al., 5 Feb 2026).

Ablations confirm that the explicit "natural" pseudo-inverse and NLBP are necessary—replacing with naïve or random inversion results in failure modes with loss of semantic consistency or divergence to noise (Ehrlich et al., 5 Feb 2026).

6. Implementation Guidelines and Operator-Theoretic Design

Critical design choices for constructing SPNNs include:

• Activation functions: Prefer pointwise bijections (e.g., LeakyReLU, strictly monotonic functions) for global invertibility/injectivity (Furuya et al., 2023).
• Layerwise operator structure: Each block should have a Fredholm operator of index zero plus compact or small-norm corrections, ensuring invertibility and surjectivity. For finite-rank truncation, output rank $g : X \to Y$8 suffices for injectivity.
• Pseudo-inverse subnetworks: For each operator, maintain a parallel subnetwork implementing (possibly local) Newton-type iteration or partition-of-unity glueing to recover global invertibility (Furuya et al., 2023).
• Stability considerations: Regularization (weight-decay, coercivity enforcement) is recommended to control contraction constants and circumvent parameter degeneracies that could break surjectivity/invertibility.
• Safety implications: Surjectivity guarantees that all outputs are reachable—this implies inherent vulnerability to adversarial input construction and motivates design of monitoring or constraint mechanisms to mitigate the risk of harmful content generation (Jiang et al., 26 Aug 2025).

The following table outlines key architectural guidance for SPNNs:

Component	Principle	Referenced Work
Activation	Use bijective (e.g., LeakyReLU)	(Furuya et al., 2023)
Layer map $g : X \to Y$9	Fredholm index 0 + compact	(Furuya et al., 2023)
Output rank (finite)	$y \in Y$0	(Furuya et al., 2023)
Pseudo-inverse process	Local Newton/partition glueing	(Furuya et al., 2023)
Surjectivity	Pre-LN, linear attn, RealNVP	(Jiang et al., 26 Aug 2025Ehrlich et al., 5 Feb 2026)

7. Limitations and Future Directions

SPNNs rely on surjectivity, which holds generically but requires avoidance of measure-zero parameter choices and attention to properness or contraction properties in infinite-dimensional settings (Jiang et al., 26 Aug 2025). The "natural" pseudo-inverse depends on the expressivity of auxiliary null-space nets; underfitting in $y \in Y$1 can produce algebraically valid but unnatural solutions (Ehrlich et al., 5 Feb 2026). Numerical inversion (gradient, fixed-point, Newton iteration) is feasible at moderate scale but suffers from slow/unstable convergence in high dimensions or near singular points. Degree-theoretic arguments assure existence, not efficient global convergence.

Future directions include:

• Application to high-fidelity physical degradations (optical, compression, ISP pipelines).
• Investigation into whether surjectivity alone endows approximate linearity properties in neural operators.
• Integration of SPNN blocks in latent diffusion models and alignment of encoder-decoder cycles to minimize artifacts relative to VAE-based approaches (Ehrlich et al., 5 Feb 2026).
• Development of robust safety certification for SPNN-based generative models, balancing completeness (reachability) with restriction of undesirable modes (Jiang et al., 26 Aug 2025).

SPNNs establish a unified theoretical and practical foundation for non-linear inversion, semantic control, and adversarial analysis in modern deep learning architectures.