Sparse Bayesian Learning Algorithms Revisited: From Learning Majorizers to Structured Algorithmic Learning using Neural Networks

Abstract: Sparse Bayesian Learning is one of the most popular sparse signal recovery methods, and various algorithms exist under the SBL paradigm. However, given a performance metric and a sparse recovery problem, it is difficult to know a-priori the best algorithm to choose. This difficulty is in part due to a lack of a unified framework to derive SBL algorithms. We address this issue by first showing that the most popular SBL algorithms can be derived using the majorization-minimization (MM) principle, providing hitherto unknown convergence guarantees to this class of SBL methods. Moreover, we show that the two most popular SBL update rules not only fall under the MM framework but are both valid descent steps for a common majorizer, revealing a deeper analytical compatibility between these algorithms. Using this insight and properties from MM theory we expand the class of SBL algorithms, and address finding the best SBL algorithm via data within the MM framework. Second, we go beyond the MM framework by introducing the powerful modeling capabilities of deep learning to further expand the class of SBL algorithms, aiming to learn a superior SBL update rule from data. We propose a novel deep learning architecture that can outperform the classical MM based ones across different sparse recovery problems. Our architecture's complexity does not scale with the measurement matrix dimension, hence providing a unique opportunity to test generalization capability across different matrices. For parameterized dictionaries, this invariance allows us to train and test the model across different parameter ranges. We also showcase our model's ability to learn a functional mapping by its zero-shot performance on unseen measurement matrices. Finally, we test our model's performance across different numbers of snapshots, signal-to-noise ratios, and sparsity levels.

• The paper demonstrates that canonical SBL algorithms are unified under a majorization-minimization framework, yielding both additive and multiplicative update rules.
• It introduces a p-SBL family and convex combinations of updates that deliver new theoretical convergence guarantees and adaptive sparse recovery performance.
• A structured neural network mimics iterative SBL updates, enabling efficient, dimension-invariant processing and improved generalization across various measurement models.

Revisiting Sparse Bayesian Learning Algorithms: Unified MM Framework and Structured Neural Architectures

Introduction

The examined paper makes significant advancements in Sparse Bayesian Learning (SBL) for sparse signal recovery in the multiple measurement vector (MMV) regime, introducing a comprehensive framework that unifies classical SBL algorithms under the majorization-minimization (MM) principle and further generalizes the algorithmic design space through structured deep learning architectures. The results provide both theoretical and practical developments in identifying, learning, and deploying SBL algorithms for diverse sparse recovery applications, addressing longstanding questions about algorithmic selection, convergence, and adaptability across problem instances.

Unified MM Framework for SBL

A primary contribution is the demonstration that canonical SBL algorithms—Expectation-Maximization SBL (EM-SBL) and Tipping's Multiplicative Update SBL (MU-SBL)—admit formulation as MM algorithms optimizing a shared surrogate (majorizer) for the SBL Type-II negative log marginal likelihood. The derivation covers both update rules, showing that EM-SBL corresponds to an additive update and MU-SBL to a multiplicative update, and both are rigorously shown to be descent steps for the same majorizer. The MM framework delivers theoretical convergence guarantees that were missing for fixed-point methods like MU-SBL. This unification is codified in the proposed $p$-SBL family of update rules, parametrized by $0 < p \leq 1$:

$$\boldsymbol{\gamma}_{j+1} = \left(\frac{\mathbf{T}_1(\boldsymbol{\gamma}_j)}{\mathbf{T}_2(\boldsymbol{\gamma}_j)}\right)^p \boldsymbol{\gamma}_j,$$

where $\mathbf{T}_1$ and $\mathbf{T}_2$ are sufficient statistics describing data- and model-dependent quantities. The convergence analysis for the entire $p$-SBL family fills a gap in the literature, furnishing new theoretical assurances for widely used SBL algorithms.

The paper advances a beamforming interpretation, connected to minimum power distortionless response (MPDR) perspectives, providing further insight: the essential difference among SBL algorithms is in how they update $\boldsymbol{\gamma}$ to match powers estimated from data versus the model. The additive (EM) or ratio (MU, $p$-SBL) update character leads to diverse convergence and recovery behavior, which in practice depends non-trivially on signal and measurement matrix structure.

Learning Majorizers and Algorithmic Expansion

The authors extend the MM-based analysis by constructing new SBL algorithms as convex combinations of majorizers and their associated update rules. The theoretical results show that, under convexity, weighted combinations of valid updates remain valid. This framework can mix different $p$-SBL rules or combine EM and $p$-SBL at each iteration, with mixing weights potentially learned from data to optimize application-specific performance. The formal results in the paper detail sufficient conditions for such combinations—e.g., convexity of pointwise minima of majorizers—to ensure MM descent properties.

A practical implication is that one can empirically optimize the sequence of majorizer (or update rule) choices to exploit the strengths of different algorithms at different stages: fast convergence initially, sharp support selection asymptotically. Empirical results confirm this, with learned weighting strategies switching from aggressive ($0 < p \leq 1$0) to conservative ($0 < p \leq 1$1) updates as iterations proceed.

Structured Neural Network Architectures for SBL

Moving beyond analytic update rules, the work introduces a novel neural architecture aligned with the MM-based reparameterization of SBL. Instead of regressing from measurements to solution in an end-to-end fashion, the network is structured to mimic iterative SBL updates: at each iteration, a small MLP consumes ($0 < p \leq 1$2) to output the next $0 < p \leq 1$3, independently for each row index $0 < p \leq 1$4. By tying the network structure and parameter sharing to the SBL problem geometry, the authors ensure:

• The network's parameter count and evaluation complexity do not depend on the measurement matrix size.
• The architecture admits generalization across measurement matrices, problem dimensions, and sparsity levels.
• Residual connections can be incorporated using fixed SBL update rules, further improving trainability and incorporating prior algorithmic knowledge.

Training employs an exponentially weighted MSE (over iterates) and support cross-entropy loss. The architecture's modularity and explanation-aligned design enable direct transfer and efficient fine-tuning on drastically different sensing matrices, including random, array manifold, and correlated dictionaries.

Numerical Results and Empirical Implications

Experiments validate the theoretical claims and neural network architecture:

• Across both array and Gaussian random matrices, classical SBL algorithms exhibit marked performance differences, illustrating the need for adaptive algorithm selection.
• Convex-combined and data-learned majorizer strategies systematically outperform any fixed SBL algorithm in both convergence speed and accuracy.
• The neural architecture—without retraining—generalizes across unseen measurement matrices and problem sizes. Fine-tuning leads to performance that matches or exceeds classical SBL algorithms across all tested conditions.
• The architecture achieves zero-shot transfer to unseen matrix types and dimensions, a direct benefit of the decoupling between architectural complexity and problem size due to the engineered input statistics ($0 < p \leq 1$5).

Theoretical and Practical Implications

By casting SBL algorithms in the MM framework, the work demystifies their convergence properties and opens principled avenues for hybrid or data-driven algorithm design. The introduction of learned majorizers and structured neural algorithmic learning bridges established signal processing methodology and modern meta-learning techniques. Practically, the resulting algorithms and architectures provide robust, dimension-invariant, and highly generalizable solutions to SSR, with clear implications for array processing, wireless channel estimation, and biomedical source localization.

Theoretically, the approach suggests a template for analyzing and extending other iterative inference algorithms—drawing connections between hand-designed updates, majorizer geometry, and learnable algorithmic “meta” parameters. The explicit unification facilitates automated algorithm selection and adaptation, informed by statistical properties of data, suggesting further research into application-optimized and uncertainty-robust variants.

Conclusion

The paper establishes a comprehensive and theoretically grounded view of SBL for sparse recovery, showing how classical and novel update rules can be unified, generalized, and further improved through both convex analytic combinations and structured neural meta-learning. The invariance of the proposed architectures to problem size, their empirical superiority, and strong theoretical guarantees suggest this approach is promising for widespread use in signal recovery tasks. Future work is warranted on extending these ideas to scenarios with model mismatch, non-Gaussianity, and application-specific transfer learning.