Differentiable SVD based on Moore-Penrose Pseudoinverse for Inverse Imaging Problems
Abstract: Low-rank regularization-based deep unrolling networks have achieved remarkable success in various inverse imaging problems (IIPs). However, the singular value decomposition (SVD) is non-differentiable when duplicated singular values occur, leading to severe numerical instability during training. In this paper, we propose a differentiable SVD based on the Moore-Penrose pseudoinverse to address this issue. To the best of our knowledge, this is the first work to provide a comprehensive analysis of the differentiability of the trivial SVD. Specifically, we show that the non-differentiability of SVD is essentially due to an underdetermined system of linear equations arising in the derivation process. We utilize the Moore-Penrose pseudoinverse to solve the system, thereby proposing a differentiable SVD. A numerical stability analysis in the context of IIPs is provided. Experimental results in color image compressed sensing and dynamic MRI reconstruction show that our proposed differentiable SVD can effectively address the numerical instability issue while ensuring computational precision. Code is available at https://github.com/yhao-z/SVD-inv.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper tackles a tricky math problem that shows up when training certain deep learning models for image reconstruction. These models often use a tool called SVD (Singular Value Decomposition) to help recover images from incomplete or noisy data. But SVD can cause training to crash when some of its numbers (called singular values) are equal or very close. The authors propose a new, safer way to compute gradients through SVD using the Moore-Penrose pseudoinverse, so training stays stable and accurate.
What questions does it ask?
In simple terms, the paper asks:
- Why does backpropagation through SVD sometimes break during training?
- Can we fix this so models that rely on SVD can train reliably?
- Will the fix keep results accurate and work well in real image tasks?
How did the researchers approach the problem?
Key ideas explained simply
- Inverse imaging problems: Imagine you’re trying to reconstruct a clear picture from blurry or incomplete data—like building a puzzle with missing pieces. This is common in areas like MRI and compressed sensing.
- Low-rank regularization and SVD: Many images or videos can be described efficiently with only a few “important directions” of information. SVD is like breaking a picture into a set of core patterns and their strengths (these strengths are the singular values). Low-rank regularization encourages the model to focus on these core patterns to reconstruct the image.
- Why SVD gradients break: Deep Unrolling Networks (DUNs) turn traditional iterative algorithms into deep networks. During training, the model needs gradients (instructions on how to improve). If two singular values are equal or extremely close, the math for the SVD gradient becomes undefined or blows up to huge numbers. This can cause numerical overflow and crash training.
- What’s really going wrong: When deriving the gradient for SVD, you end up solving a system of linear equations. If singular values repeat, that system becomes underdetermined—meaning there isn’t a unique solution. The gradient is not well-defined, which causes instability.
- The fix: Use the Moore-Penrose pseudoinverse. Think of it as a method that finds the “best possible” solution when there isn’t a unique one. By applying the pseudoinverse to the broken linear system, the authors get a stable, minimum-norm solution for the SVD gradient. They call this approach “SVD-inv.”
- Making it stable in practice: The authors also analyze which parts of the gradient could blow up and apply safe thresholds. If a computed value gets too large, they treat it as a special case, move it to a safe term, and cap it. This keeps training within safe numeric limits (especially with Float32, the common precision used in deep learning).
What did they find?
- Theory: They show clearly that non-differentiability of SVD happens because the gradient derivation leads to an underdetermined system when singular values repeat. Using the pseudoinverse gives a principled, stable gradient.
- Numerical stability: They provide a step-by-step analysis showing where instability arises and how their method keeps all values within safe ranges, even when singular values are equal or very close.
- Experiments: They test their method in two real tasks:
- Color image compressed sensing (reconstructing images from very few measurements)
- Dynamic MRI (reconstructing a sequence of medical images over time)
In both tasks, their SVD-inv method: - Prevented training crashes caused by unstable SVD gradients - Kept gradient calculations accurate (lower errors compared to other fixes like Taylor expansions or clipping) - Improved overall performance and reliability of the deep unrolling networks
- Comparison with other fixes: Methods like Taylor approximations, clipping large values, or library defaults (e.g., setting infinities to zero) either distort gradients or don’t fix the root cause. SVD-inv is grounded in solid math and consistently produced smaller gradient errors.
Why does this matter?
- More reliable training: Models that use SVD for low-rank regularization can now train stably without mysterious failures.
- Better image reconstructions: In areas like medical imaging (MRI) or compressed sensing, stable training can lead to clearer results, faster experiments, and more robust systems.
- General applicability: The approach also applies to positive semi-definite matrices (a common case where SVD stands in for eigenvalue decomposition), so it can benefit a wide range of tasks beyond imaging.
Takeaway and potential impact
This work provides a mathematically sound and practical solution to a common training instability in deep learning models that rely on SVD. By using the Moore-Penrose pseudoinverse to fix the underdetermined gradient problem, the authors make deep unrolling networks more dependable and accurate for inverse imaging tasks. This can help researchers and engineers build better tools for medical imaging, video reconstruction, and other applications where you need to recover high-quality images from limited or noisy data.
Code for the method is available at: https://github.com/yhao-z/SVD-inv
Collections
Sign up for free to add this paper to one or more collections.