2000 character limit reached
Pseudo-Invertible Neural Networks
Published 5 Feb 2026 in cs.LG and cs.CV | (2602.06042v1)
Abstract: The Moore-Penrose Pseudo-inverse (PInv) serves as the fundamental solution for linear systems. In this paper, we propose a natural generalization of PInv to the nonlinear regime in general and to neural networks in particular. We introduce Surjective Pseudo-invertible Neural Networks (SPNN), a class of architectures explicitly designed to admit a tractable non-linear PInv. The proposed non-linear PInv and its implementation in SPNN satisfy fundamental geometric properties. One such property is null-space projection or "Back-Projection", $x' = x + A\dagger(y-Ax)$, which moves a sample $x$ to its closest consistent state $x'$ satisfying $Ax=y$. We formalize Non-Linear Back-Projection (NLBP), a method that guarantees the same consistency constraint for non-linear mappings $f(x)=y$ via our defined PInv. We leverage SPNNs to expand the scope of zero-shot inverse problems. Diffusion-based null-space projection has revolutionized zero-shot solving for linear inverse problems by exploiting closed-form back-projection. We extend this method to non-linear degradations. Here, "degradation" is broadly generalized to include any non-linear loss of information, spanning from optical distortions to semantic abstractions like classification. This approach enables zero-shot inversion of complex degradations and allows precise semantic control over generative outputs without retraining the diffusion prior.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper tackles a big question in math and AI: how to “undo” a process when that process throws away some information. In simple terms, it builds special neural networks that can shrink or simplify data (like turning a photo into a short description) but still come with a reliable way to go back and get a sensible, full version again. The authors call these networks Surjective Pseudo-invertible Neural Networks (SPNNs), and they introduce a new way to “back-project” results so they match real measurements even when the process is non-linear (not just a straight line or simple equation).
What goals or questions does it try to answer?
The paper focuses on three main goals:
- Create a “natural” version of the pseudo-inverse for non-linear functions (a smart way to undo things when perfect reversal is impossible).
- Design a neural network architecture (SPNN) that naturally includes this pseudo-inverse, even when it reduces the size of the data.
- Use a new method called Non-Linear Back-Projection (NLBP) to guide powerful image generators (like diffusion models) so their outputs match real-world measurements or chosen attributes—without retraining those generators.
How did the authors approach the problem?
Think of “invert” as “undo.” In math, the classic Moore–Penrose pseudo-inverse is a way to undo linear transformations that might lose information, finding the best possible solution. But modern neural networks are usually non-linear, making the old trick not directly usable. The authors generalize the idea:
- A “surjective” function is one where every possible output has at least one input—but inputs can map to the same output, so some info gets lost.
- To pick a unique, sensible “undo” among many possible ones, the authors add extra coordinates (a “bijective completion”) that make the combined mapping reversible. Then they choose the solution that’s closest to the center in this completed space. Intuition: they add a smart set of extra tags so the function becomes reversible, and they pick the most “neutral” undo.
They build SPNNs around this idea:
- Each SPNN block splits the input into two parts: one part carries the “kept” content, the other part acts like “extra knobs” that influence the kept content.
- The forward pass compresses the input (reducing dimensions) using learned scale and shift functions.
- The pseudo-inverse uses an auxiliary network to predict the missing “extra knobs” from the compressed output, then reverses the coupling to reconstruct a full, high-dimensional version.
They also introduce Non-Linear Back-Projection (NLBP):
- In the linear world, “back-projection” nudges your current guess so it exactly matches measured data while staying as close as possible to your guess.
- NLBP does the same for non-linear mappings using the SPNN’s structure, guaranteeing the output matches the measurement and, in their setup, is the closest valid fix.
Finally, they plug NLBP into pretrained diffusion models (popular image generators). During sampling, NLBP gently steers the generated image so its measured attributes (like “Smiling”, “Eyeglasses”) match a target, while leaving fine details intact.
What did they find and why does it matter?
- The SPNN architecture has a built-in pseudo-inverse that works even when the network compresses data. This isn’t true for standard invertible networks, which must keep the same size.
- NLBP guarantees that after a correction step, the output matches the desired measurement (like a classification vector) and, in their “natural” setup, the fix is the smallest possible change from the current image.
- In tests on face images (CelebA‑HQ), they reconstructed realistic images from just 40 semantic attributes (like “Male”, “Smiling”, “Eyeglasses”). On average, their reconstructions agreed with the target attributes about 92% of the time.
- They showed powerful attribute control: they could guide a diffusion model to produce faces that, for example, definitely have glasses or combine multiple attributes (e.g., “Male + Eyeglasses + Smiling”) without retraining the generator.
- Ablation studies (turning off key parts) failed badly, showing the “natural” pseudo-inverse and careful NLBP are essential.
Why it matters: Many real-world problems involve non-linear “degradations” (like camera pipelines, compressions, or even turning an image into a label). This work gives a principled, math-backed way to reverse or control such processes without retraining huge generative models. It bridges classic signal processing ideas with modern deep learning.
What are the implications and potential impact?
- Practical benefits: Better “zero-shot” restoration (fixing or recreating data without retraining), and precise control over what generative models produce (e.g., setting chosen attributes in images).
- Scientific impact: A clear, rigorous path to generalize the beloved linear pseudo-inverse to non-linear neural networks, keeping core consistency guarantees.
- Limitations and care: The auxiliary network that predicts missing info must be good enough; otherwise, the “undo” may be technically correct but look unrealistic. The approach assumes the forward mapping really can produce your target measurement. Also, as with any strong generative control, there are risks (deepfakes, biases from training data).
- Future directions: Applying SPNNs to more complex measurement processes (like object detectors or optics), using surjective ideas to design more efficient “linearizer” modules, and fixing cycle-consistency issues in common generative pipelines.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The following list summarizes concrete gaps and unresolved questions that future work should address to strengthen, generalize, and validate the proposed framework.
- Existence and construction of bijective completion G: Formal conditions under which a surjective continuous operator g admits a bijective (ideally diffeomorphic) completion G(x) = [g(x) | q(x)]T, and practical methods to construct q for arbitrary g beyond the SPNN architecture.
- Uniqueness of the minimizer in the completed space: Precise conditions guaranteeing that argmin_{x∈g{-1}(y)} ||G(x) − G(0)|| has a unique minimizer (e.g., convexity, regularity of G), and what happens when multiple minimizers exist.
- Choice of origin G(0) and invariance: How sensitive the “Natural” PInv is to the choice of origin in the completed space; whether a translation-invariant or data-driven center (e.g., mean of q(x)) yields more canonical solutions, and how to standardize this choice.
- Metric dependence and geometry of the latent space: Justification for using Euclidean inner products in the G-latent space, and whether alternative metrics (e.g., learned Riemannian metrics, whitening) improve orthogonality and projection properties.
- Theoretical relation to BAS/minimal-norm PInv: Formal characterization of when the proposed PInv (via bijective completion) coincides with the Best Approximate Solution (BAS) approach (Gofer & Gilboa, 2023) in non-linear settings; counterexamples and equivalence conditions.
- Continuity, Lipschitzness, and stability of gt: Bounds on the continuity and Lipschitz constants of gt with respect to y and to perturbations in g; sensitivity analysis under small changes or noise in measurements.
- Handling inconsistent or noisy measurements: Extension of NLBP and the PInv to the case y ∉ Range(g) or y corrupted by noise; derivation of regularized updates (e.g., Tikhonov-like variants) and convergence guarantees in the noisy regime.
- Convergence guarantees for NLBP in practice: Formal convergence analysis of the iterative NLBP update (alone and when combined with diffusion sampling), including step-size schedules and conditions for reaching the closest pre-image in the G-metric.
- Impact of approximation error in g: Quantitative analysis of how modeling D(x) with an approximate g (SPNN) affects NLBP consistency and reconstruction quality; error propagation bounds from g to gt to the final projection.
- Existence of diffeomorphic G under typical neural nonlinearities: Clarification on whether SPNNs (with ReLU, tanh, exp, affine couplings, and dimensionality reduction) actually produce a G that is diffeomorphic, and what modifications are required to satisfy Definition 4.1 rigorously.
- Surjectivity assumptions on real degradations: Verification of surjectivity for practical operators (classifiers/logit maps, ISPs, JPEG) and characterization of their reachable ranges; procedures for diagnosing and adapting when g is not surjective.
- Auxiliary network r expressivity and identifiability: Criteria for the capacity and training of r to reliably learn the null-space statistics; diagnostics for failure modes (e.g., producing unrealistic pre-images) and architectural remedies.
- Multi-block and multi-scale composition of G: Formal definition of G for deep, multi-block SPNNs; how q’s at different scales compose; conditions ensuring overall surjectivity and a well-defined global PInv.
- Null-space dimensionality vs. recoverability: Empirical and theoretical study of how the size of the discarded subspace (D−d) affects the feasibility of reconstructing plausible pre-images from y using r (and/or a generative prior).
- Robustness to out-of-distribution inputs: Evaluation of SPNN/NLBP performance when y or x0|t are out-of-distribution, and strategies (e.g., covariance-aware adjustments, OOD detectors) to maintain plausibility.
- Generalization beyond facial semantics: Validation on diverse, non-linear degradations (e.g., ISP pipelines, compression artifacts, optical aberrations, object detection outputs, captions) to demonstrate breadth beyond CelebA-HQ attributes.
- Comparative baselines and metrics: Systematic comparisons with DPS, PnP-Diffusion, and other non-linear guidance methods on shared benchmarks with quantitative metrics (e.g., FID, LPIPS, semantic accuracy, calibration curves).
- Interaction with diffusion sampling correctness: Analysis of how NLBP guidance affects the marginal distribution induced by the diffusion model; trade-offs between fidelity and distributional integrity, and principled schedules for λ.
- Scheduling and step-size design: Formal study of when to activate NLBP within the diffusion trajectory, how to set λ adaptively, and guarantees that guidance does not destabilize early noisy stages.
- Role of orthogonal mixing U: Quantification of how Cayley-parameterized mixing impacts separation of content vs. redundancy, and whether learned rotations improve PInv quality; ablations and theory linking U to identifiability.
- Architectural and loss ablations for r: Systematic exploration of r architectures, Lnatural weights, and auxiliary losses (surjectivity/stability) to determine minimal configurations that still yield a reliable PInv.
- Scalability and efficiency: Complexity analysis for training and inference in SPNNs (especially with high-dimensional x and deep stacks), memory footprints, and performance in large-scale settings.
- Bias and fairness in canonical solutions: Empirical quantification of demographic or attribute biases encoded by r and G; methods to debias or constrain the “Natural” PInv to avoid amplifying dataset biases.
- Extensions to stochastic operators: Handling degradations with inherent randomness (e.g., dropout, noise-injection pipelines) where g is not deterministic; defining PInvs and NLBP variants for stochastic mappings.
- Probabilistic PInv and multi-modal pre-images: Frameworks for representing and sampling the distribution over pre-images (rather than a single “natural” solution), and integrating such uncertainty into restoration objectives.
Practical Applications
Overview
Below are concrete, real-world applications enabled by the paper’s contributions: Surjective Pseudo-Invertible Neural Networks (SPNN), the “Natural” non-linear pseudo-inverse, and Non-Linear Back-Projection (NLBP). Each item notes who benefits (industry, academia, policy, daily life), the sector(s), potential tools/products/workflows, and the key assumptions/dependencies affecting feasibility.
Immediate Applications
These can be piloted or deployed with today’s models and infrastructure, assuming access to data to train an SPNN surrogate for a target operator and an off-the-shelf diffusion prior.
- Bold semantic control in diffusion without retraining
- Audience: Industry (creative tools, media), Daily life
- Sectors: Software, Media/Entertainment, Design
- What: Add fine-grained attribute control (e.g., “eyeglasses,” “smile,” “blond hair”) to existing diffusion pipelines by inserting the NLBP step guided by an SPNN that approximates a semantic classifier.
- Tools/workflows: Plugins for Stable Diffusion/ComfyUI/Automatic1111 that wrap an SPNN module and expose slider-based attribute controls; API endpoints for “classifier-consistent” generation; batch guidance scripts for studios.
- Assumptions/dependencies: A pre-trained diffusion model; a reliable classifier and sufficient (image, attribute) data to train the SPNN; surjectivity (every target attribute vector must be realizable); compute budget for guidance steps.
- Zero-shot restoration of non-linear degradations
- Audience: Industry (photography, mobile, imaging), Daily life
- Sectors: Computational Photography, Consumer Devices, Software
- What: Restore images degraded by non-linear pipelines (e.g., ISP, tone mapping, gamma, JPEG, denoisers) by training an SPNN surrogate of the black-box degradation and inserting NLBP into the diffusion sampler.
- Tools/workflows: Camera/phone firmware or app post-processing “non-linear deartifacting” mode; Photoshop/GIMP filters for non-linear artifact removal; developer SDKs that provide SPNN+NLBP wrappers for legacy image pipelines.
- Assumptions/dependencies: Access to representative input-output pairs from the target degradation to fit the SPNN; degradation should be well-approximable and surjective over the operating set; latency constraints for on-device use.
- Classifier-consistent reconstruction and debugging for model QA
- Audience: Industry (ML Ops), Academia
- Sectors: Software/AI Dev, Responsible AI
- What: Given classifier logits, generate the closest input on the data manifold that matches them (via NLBP), enabling sanity checks (“What does this model consider a ‘smiling’ face?”), error analysis, and dataset auditing.
- Tools/workflows: QA dashboards that visualize canonical pre-images per class/attribute; unit tests that detect spurious shortcuts; curriculum for interpretability labs.
- Assumptions/dependencies: Trained SPNN that accurately approximates the classifier mapping; diffusion prior aligned with dataset domain; governance on use to avoid privacy violations.
- Black-box system inversion via surrogate modeling
- Audience: Industry (hardware vendors, imaging), Academia
- Sectors: Imaging, AR/VR, Consumer Electronics
- What: Fit an SPNN to a proprietary or opaque forward process (e.g., denoise-tonemap pipeline, lens shading) and use NLBP to enforce measurement consistency when enhancing images or textures.
- Tools/workflows: Hardware-in-the-loop calibration pipelines; lab bench “inversion harness” for ISP validation; SDK for OEMs to prototype inverse operators without exposing IP.
- Assumptions/dependencies: Sufficient observability (I/O data) for the forward process; stability under domain shift; surjectivity across intended operating conditions.
- Measurement-consistent generation as a diffusion add-on
- Audience: Industry (GenAI platform providers), Academia
- Sectors: Software, ML Infrastructure
- What: A general guidance module that enforces non-linear measurement consistency (SPNN+NLBP) in any diffusion sampler, analogous to linear null-space projection.
- Tools/workflows: Reusable PyTorch/JAX library exposing “G-aware” projection hooks; inference-time flags (e.g., --nlbp spnn.ckpt --lambda 0.2) for samplers.
- Assumptions/dependencies: A bijective completion G implemented by the SPNN; careful guidance scheduling; compute overhead acceptable for product SLAs.
- Instructional and research tooling for non-linear pseudo-inversion
- Audience: Academia
- Sectors: Education, ML Theory, Signal Processing
- What: Teach generalized inverses beyond the linear case; benchmark suites comparing Gofer–Gilboa BAS vs. SPNN’s bijective-completion PInv; reproducible labs on non-linear back-projection.
- Tools/workflows: Open-source SPNN blocks, training recipes, and NLBP samplers; course modules and notebooks for inverse problems.
- Assumptions/dependencies: Availability of clean baselines; datasets with known ground-truth forward operators.
- Risk and robustness testing for generative systems
- Audience: Policy, Industry (Trust & Safety)
- Sectors: Responsible AI, Security
- What: Stress-test how easily semantics can be steered in generative systems without retraining; evaluate exposure to misuse (e.g., attribute fabrication) and inform guardrails.
- Tools/workflows: Internal red-teaming harnesses that apply NLBP-based steering to test content policies; monitoring thresholds for semantic drift.
- Assumptions/dependencies: Organizational processes for model governance; clear policies on acceptable attribute controls; detection and logging.
- Robotics/computer vision pre-processing under non-linear optics
- Audience: Industry (Robotics, Drones, ADAS)
- Sectors: Robotics, Automotive
- What: Use SPNN to approximate lens/ISP non-linearities and apply NLBP to project sensor frames onto a “consistent” pre-image before downstream perception, reducing domain gap.
- Tools/workflows: Perception stacks that insert an SPNN+NLBP normalization stage; calibration datasets from varied lighting/lenses.
- Assumptions/dependencies: Stationary or slowly varying optics; compute budget on edge devices; known operating envelope for surjectivity.
Long-Term Applications
These require further research, scaling, or integration (e.g., domain-specific data, regulatory approval, real-time constraints).
- Medical imaging inversion with non-linear acquisition models
- Audience: Healthcare, Academia
- Sectors: Medical Imaging (MRI, CT, Ultrasound)
- What: Enforce data fidelity for non-linear measurement chains (e.g., coil compression, non-linear beamforming, dose-aware denoising) using SPNN surrogates and NLBP to reconstruct diagnostically faithful images.
- Tools/products: Reconstruction modules integrated into PACS or scanner software; research prototypes for non-linear MR/US inverse solvers.
- Assumptions/dependencies: Clinically representative training data; rigorous validation; regulatory approval; safety, bias, and failure-mode audits; clear surjectivity over inputs of interest.
- Inversion of high-level perception (object detection/captioning) for controllable synthesis
- Audience: Industry (content creation), Education, Research
- Sectors: Media/Entertainment, EdTech
- What: Train SPNNs on detectors/captioners to reconstruct or generate scenes from sparse semantics (bounding boxes, captions) with precise consistency, enabling fast layout-to-image or caption-consistent generation.
- Tools/products: “Compose-by-constraints” creative tools; storyboard-to-shot pipelines; educational tools that visualize semantics-to-image mappings.
- Assumptions/dependencies: Large-scale paired (image, semantics) datasets; managing ambiguity and multimodality; fairness and representation balance.
- SPNN encoders for latent diffusion and autoencoders
- Audience: Software/ML Platforms, Academia
- Sectors: ML Infrastructure, Generative AI
- What: Replace VAE encoders with SPNN blocks to improve cycle-consistency and measurement fidelity, reducing entanglement and posterior mismatch in latent diffusion systems.
- Tools/products: Next-gen latent diffusion models with pseudo-invertible encoders; training frameworks that jointly optimize forward tasks and natural inverses.
- Assumptions/dependencies: Stable training of deep SPNNs; careful choice of bijective completion; empirical gains over strong VAE baselines.
- Real-time state estimation and sensor fusion under non-linear measurement in robotics
- Audience: Industry (Robotics, Autonomous Systems)
- Sectors: Robotics, Aerospace, Automotive
- What: Embed SPNN-based inverses as consistency layers in SLAM/visual-inertial odometry and fusion pipelines to handle non-linear sensor models (e.g., rolling shutter, non-linear IMU biases).
- Tools/products: Middleware libraries for ROS/Autoware; embedded accelerators for SPNN inference.
- Assumptions/dependencies: Deterministic latency; robust generalization; training data covering edge cases; certifiable safety.
- Scientific computing and digital twins: inverse modeling for non-linear simulators
- Audience: Energy, Manufacturing, Climate, Academia
- Sectors: Energy, Engineering, Geoscience
- What: Surrogate SPNNs of complex non-linear forward simulators (e.g., turbulence closures, grid sensors with non-linearities) to perform measurement-consistent inverse inference guided by priors.
- Tools/products: Inverse solvers in digital twin platforms; HPC toolkits coupling SPNN surrogates with diffusion priors for posterior exploration.
- Assumptions/dependencies: High-fidelity simulation/measurement pairs; quantification of uncertainty; interpretability and auditability for high-stakes contexts.
- Telecommunications and network tomography with non-linear channels
- Audience: Telecoms, Academia
- Sectors: Communications
- What: Invert non-linear channel effects (e.g., compression, quantization, non-linear amplifiers) for diagnostics or reconstruction using SPNN+NLBP.
- Tools/products: Network monitoring and diagnostics platforms; baseband firmware enhancements.
- Assumptions/dependencies: Access to channel I/O traces; stationarity or adaptive retraining strategies; integration with existing PHY stacks.
- Finance and econometrics: consistent input reconstruction for non-linear risk/valuation models
- Audience: Finance, Academia
- Sectors: Finance, Risk Analytics
- What: Recover plausible inputs consistent with target risk scores or model outputs to stress-test and interpret non-linear models (e.g., deep credit scorers).
- Tools/products: Model validation toolkits that generate canonical pre-images for given outputs; scenario generators constrained by model outputs.
- Assumptions/dependencies: Strict controls to avoid privacy leakage; domain-appropriate priors; governance frameworks.
- Bias auditing and policy frameworks for “canonical” pre-images
- Audience: Policy, Responsible AI teams
- Sectors: Governance, Ethics
- What: Use canonical reconstructions to audit whether non-linear models encode demographic stereotypes (e.g., how attributes co-occur in reconstructions), informing standards and disclosures.
- Tools/products: Audit reports and probes that evaluate attribute co-variation; compliance checkpoints for generative deployments.
- Assumptions/dependencies: Diverse, representative training data; clear metrics for bias; stakeholder review; safeguards against misuse.
- Accessibility: attribute-to-image assistive synthesis
- Audience: Daily life, Non-profits, Education
- Sectors: Accessibility, Assistive Tech
- What: Convert structured attributes or descriptions into consistent images (e.g., educational visualizations for low-vision users), with controllable semantics.
- Tools/products: Assistive apps that render scenes from attribute layouts; educational content generators aligned to curricula.
- Assumptions/dependencies: Safety filters; cultural sensitivity; robust control over harmful content; consent and privacy considerations.
Cross-Cutting Assumptions and Dependencies
- Surjectivity and coverage: The forward operator being approximated (e.g., classifier, ISP) must be surjective over the target domain; measurements y should lie within its range to guarantee consistent inversion.
- Quality of the surrogate: SPNN must closely approximate the forward operator; auxiliary network r must be expressive enough to learn the null-space statistics to yield realistic pre-images.
- Prior alignment: The diffusion prior must reflect the same data distribution as the forward operator; domain shift degrades results.
- Compute and latency: NLBP adds inference-time overhead; real-time or edge deployments require optimization or accelerators.
- Data requirements: Training SPNNs requires paired (input, measurement) data; for proprietary pipelines, data collection and consent are prerequisites.
- Safety and ethics: Reconstruction from sparse semantics can enable misuse (e.g., deepfakes); deployments need safeguards, logging, and policy compliance.
- Numerical stability: Deep SPNN stacks benefit from stability losses and careful scheduling of guidance strength, as described in the paper.
These applications collectively translate the paper’s theoretical and architectural advances into practical tools and workflows across creative industries, imaging, robotics, scientific computing, finance, and governance, while highlighting when further research or infrastructure is required.
Glossary
- Adjoint (linear adjoint): For linear operators, the adjoint (conjugate transpose) underlies orthogonality and projection properties; it is not generally defined for non-linear mappings. "the final two Penrose identities, however, are inapplicable in the non-linear setting as they rely on the linear adjoint."
- Affine coupling: A transformation where part of the input modulates the other via learned element-wise scale and shift, enabling tractable invertibility in flow models. "The forward operation g(.) is defined as an affine coupling, where x1 modulates x0 via learned scale and translation functions:"
- Back-projection kernel: A heuristic operator approximating the pseudo-inverse to back-project measurement errors during iterative reconstruction. "where H is a back-projection kernel (approximating At) and X is a step size."
- Best Approximate Solution (BAS): A principle defining the non-linear pseudo-inverse in metric spaces as the right-inverse that recovers the minimal-norm element of the pre-image. "proposing that the unique "pseudo-inverse" in metric spaces be defined by the Best Approximate Solution (BAS) property,"
- Bijective Completion: An augmentation that turns a surjective operator into a bijection by adding auxiliary coordinates so the combined mapping becomes invertible. "A Bijective Completion is a diffeomorphism G : X > > x Z defined as:"
- Cayley transform: A parameterization that maps skew-symmetric matrices to orthogonal/unitary matrices, used to ensure exact orthogonality in learnable rotations. "parameterized via the Cayley transform to ensure exact orthogonality (UTU = I) by construction."
- Conditional diffusion: A generative paradigm where diffusion models are conditioned on observations or labels for guided sampling or inversion. "In contrast, "inversion" in modern Deep Learning is typically approached either as a regression task (autoencoders) or a probabilistic generation task (conditional diffusion)."
- DDNM: Denoising Diffusion Null-space Model; a diffusion-based approach to linear inverse problems that exploits null-space projections. "Null-Space Methods (Linear): Approaches like DDRM (Kawar et al., 2022), DDNM (Wang et al., 2023), and SNIPS (Kawar et al., 2021)"
- DDPM: Denoising Diffusion Probabilistic Model; a generative model with a forward noise process and learned reverse denoising dynamics. "We utilize a pre-trained unconditional DDPM with T = 1000 timesteps."
- DDRM: Denoising Diffusion Restoration Models; methods that use diffusion priors with closed-form linear back-projection for restoration. "Null-Space Methods (Linear): Approaches like DDRM (Kawar et al., 2022), DDNM (Wang et al., 2023), and SNIPS (Kawar et al., 2021)"
- Diffeomorphism: A smooth, invertible map with a smooth inverse between manifolds, used to define coordinate transformations. "A Bijective Completion is a diffeomorphism G : X > > x Z defined as:"
- Diffusion Posterior Sampling (DPS): A guidance technique that backpropagates gradients of posterior objectives through diffusion models for non-linear inverse problems. "Methods like DPS (Diffusion Posterior Sampling) (Chung et al., 2023) and PnP-Diffusion (Chung et al., 2022) handle non-linear operators"
- ELBO (Evidence Lower Bound): A variational objective that lower-bounds log-likelihood, used for training probabilistic generative models. "whereas SurVAE optimizes a stochastic lower bound on the likelihood (ELBO) (Nielsen et al., 2020),"
- Feistel Cipher: A reversible block cipher structure that splits data into halves and applies round functions to achieve exact invertibility. "with the Feistel Cipher (Feistel, 1973), which introduced the concept of splitting data into halves and mod- ifying one half conditioned on the other to ensure exact reversibility."
- Generalized inverse: An operator that acts as an inverse when a mapping is not bijective, restoring solvability or consistency. "The concept of a generalized inverse is foundational to sig- nal processing and data analysis."
- Glow: A normalizing flow architecture that introduced invertible 1x1 convolutions for learnable channel mixing. "Glow (Kingma & Dhariwal, 2018), which introduced invertible 1x1 con- volutions to replace fixed channel permutations with learn- able unitary mixing,"
- Hermitian: A matrix equal to its conjugate transpose; Hermitian projectors ensure orthogonality and least-squares properties. "Equations (5) and (6) enforce that the projection operators AAt and At A are orthogonal (Hermitian),"
- Invertible Neural Networks (INNs): Neural architectures designed for exact reversibility, often constrained to preserve dimensionality. "architectures that do enforce rigorous invertibility, such as Invertible Neural Networks (INNs) and Normalizing Flows, are constrained by strict bijectivity."
- Iterative Back-Projection (IBP): An algorithm that iteratively adds back-projected error to enforce measurement consistency in reconstructions. "Irani and Peleg (Irani & Peleg, 1991) formalized this with the Iterative Back-Projection (IBP) algorithm."
- Jacobian determinant: The determinant of the Jacobian matrix of a transformation; controlling it enables tractable likelihood in flows. "which proposed additive coupling layers to enforce a tractable Jacobian determinant of unity."
- Latent space: An internal representation space where the model analyzes components and performs projections. "We analyze the components of this displacement in the latent space of G."
- Left inverse: A map that recovers inputs when composed on the left with the forward map, existing for injective operators. "This allows for a Left Inverse 971 : y -> X satisfying:"
- Manifold: A smooth space; the set of solutions satisfying a constraint (e.g., g-1(y)) forms a manifold used for projections. "Any tangent vector v to the solution manifold (where g(z) = y is constant)"
- MMSE (Minimum Mean Square Error): The optimal least-squares estimator minimizing expected squared error. "providing the optimal least-squares (MMSE) approximation when no exact solution exists."
- Minimal-norm solution: Among all consistent solutions, the one that minimizes a chosen norm (typically Euclidean). "Uniquely, the PInv guarantees the minimal-norm solution among all valid inverses,"
- Moore-Penrose Pseudo-inverse (PInv): The canonical generalized inverse for linear operators that satisfies the Penrose identities. "The Moore-Penrose Pseudo-inverse (PInv) serves as the fundamental solution for linear systems."
- NICE (Non-linear Independent Components Estimation): An early invertible architecture employing additive coupling layers. "NICE (Non-linear Independent Components Estima- tion) (Dinh et al., 2014), which proposed additive coupling layers"
- Non-Linear Back-Projection (NLBP): A projection method that enforces consistency for non-linear surjective mappings via a non-linear pseudo-inverse. "We introduce Non-Linear Back-Projection (NLBP), which projects inputs onto the pre-image of non-linear surjective mappings."
- Normalizing Flows: Invertible transformations with tractable Jacobians used for exact density modeling and generative sampling. "Invertible Neural Networks (INNs) and Normalizing Flows, are constrained by strict bijectivity."
- Null-Space Methods: Linear inverse techniques that exploit SVD-based projections to enforce measurement fidelity. "Null-Space Methods (Linear): Approaches like DDRM (Kawar et al., 2022), DDNM (Wang et al., 2023), and SNIPS (Kawar et al., 2021)"
- Null-space projection: Moving within the null-space to reach the closest state that satisfies measurement constraints. "One such property is null-space projection or "Back-Projection", x' = x + At(y - Ax),"
- Orthogonal mixing: Preceding coupling splits with learnable orthogonal/unitary transforms to better separate content and redundancy. "Orthogonal Mixing. Standard coupling layers operate on fixed channel splits (e.g., splitting the first k channels)."
- Orthogonal projection: A projection that minimizes distance to the constraint set by being perpendicular to its tangent space. "for this update to act as an orthogonal projection,"
- Penrose identities: Four algebraic conditions defining the Moore-Penrose pseudo-inverse and its projector properties. "it is formally defined by the four Penrose Identities (Penrose, 1955)."
- PixelUnshuffle: A downsampling operation that rearranges pixels into channels to create multi-scale representations. "We utilize the PixelUnshuffle operation to trade spatial resolution for channel depth,"
- PnP-Diffusion: Plug-and-Play diffusion guidance that backpropagates errors through a forward model during sampling. "Methods like DPS (Diffusion Posterior Sampling) (Chung et al., 2023) and PnP-Diffusion (Chung et al., 2022) handle non-linear operators"
- Pre-image: The set of inputs mapping to a given output under a function. "projects inputs onto the pre-image of non-linear surjective mappings."
- RealNVP: A normalizing flow model using affine coupling to improve expressivity while maintaining tractable inversion. "This was subsequently improved by RealNVP (Dinh et al., 2017), which utilized affine coupling layers (incorporat- ing scaling) to enhance expressivity,"
- Reflexive Consistency: The property that composing a map with its pseudo-inverse returns consistent inputs/outputs. "Satisfying these two identities guarantees Reflexive Consis- tency,"
- Right inverse: A map that recovers outputs when composed on the right with the forward map, existing for surjective operators. "The op- erator gt defined in Eqs. (19)-(20) is a strict right-inverse of g, satisfying g(gt (y)) = y"
- SNIPS: A diffusion-based approach to stochastic solving of noisy inverse problems via null-space projection. "Null-Space Methods (Linear): Approaches like DDRM (Kawar et al., 2022), DDNM (Wang et al., 2023), and SNIPS (Kawar et al., 2021)"
- SVD (Singular Value Decomposition): A matrix factorization used to compute pseudo-inverses and measurement subspaces. "While typically computed via SVD,"
- Sur VAE Flows: Surjective flow layers bridging the gap between VAEs and flows by allowing dimension-reducing operations. "Sur VAE Flows (Nielsen et al., 2020) introduced a framework of surjective layers (e.g., pooling, slicing) to bridge the gap between VAEs and Flows."
- Surjective: A mapping whose range covers the entire target space; guarantees existence of a right inverse. "a class of architectures designed to model surjective mappings where the input dimensionality ex- ceeds the output,"
- Surjective Pseudo-invertible Neural Networks (SPNN): Dimension-reducing architectures with a built-in non-linear pseudo-inverse satisfying reflexive identities. "We introduce Surjective Pseudo-invertible Neural Net- works (SPNN),"
- Unitary matrix: A matrix with orthonormal columns/rows (UTU = I), used for stable, exact rotations and mixing. "a learnable unitary matrix U, parameter- ized via the Cayley transform to ensure exact orthogonality (UTU = I) by construction."
- Unitary mixing: Learnable unitary transformations that mix channels while preserving orthogonality. "learn- able unitary mixing,"
Collections
Sign up for free to add this paper to one or more collections.