Locally Spike Sparse (LSS) Model

• The Locally Spike Sparse (LSS) model is a framework that detects locally active features and their interactions using threshold-based rules and structured spike-and-slab priors.
• It leverages decision mechanisms in random forests and hierarchical Bayesian regressions to recover interpretable local support with provable recovery guarantees.
• The model employs rigorous mathematical assumptions and efficient inference methods, such as Expectation Propagation, to enable scalable applications in complex supervised learning tasks.

The Locally Spike Sparse (LSS) model provides an analytical and inferential framework for identifying the relevance of features and feature interactions in complex supervised learning setups, particularly under local and structured sparsity regimes. LSS characterizes scenarios in which specific combinations of features are locally active, revealing decision mechanisms for both axis-aligned logical rules and structured regression surfaces. The formulation is central to model-specific interpretability (e.g., for random forests) and hierarchical Bayesian sparse regression, accommodating both piecewise-constant logic (via signed indicator rules) and probabilistic spike-and-slab shrinkage mechanisms with structured priors.

1. Model Specification and Generative Structure

The LSS data-generating process is formalized for i.i.d. samples $D=\{(x_1,y_1),\dots,(x_n,y_n)\}$, where $x_i\in[0,1]^p$ denotes the feature vector for the $i$th sample and $y_i\in \mathbb{R}$ the associated outcome. The conditional response function given features $x$ is modeled as

$$(Y|X=x) = \beta_0 + \sum_{j=1}^J \beta_j \prod_{k\in S_j} \mathbf{1}(x_k \le \gamma_k)$$

where each $S_j \subset \{1,\dots,p\}$ defines a basic interaction (a set of feature indices), coefficients $\beta_j > C_\beta > 0$ capture the signal strength, and thresholds $\gamma_k \in (C_\gamma, 1-C_\gamma)$ (with $C_\gamma \in (0, 1/2)$) specify activation. Crucially, the $S_j$ are pairwise disjoint, ensuring non-overlapping basic interactions. This construction yields a model in which the local response is dictated by the joint presence of "spikes" (active features crossing thresholds) restricted to axis-aligned hyperrectangles in input space (Vuk et al., 11 Dec 2025).

Alternatively, in Bayesian sparse linear regression, LSS appears as a hierarchical spike-and-slab prior modulated by local spatio-temporal structure. For regression weights $w=(w_1,\dots,w_N)^\top$ and Gaussian likelihood $p(y|X,w,\sigma^2)=\mathcal{N}(y|Xw,\sigma^2 I)$, each weight's activity is switched by a spike indicator $z_i \in \{0,1\}$, governed by a latent Gaussian process $f\sim\mathcal{N}(f|0,K)$ and Bernoulli links $z_i|f_i \sim \text{Bernoulli}(\pi_i)$ with $\pi_i=\Phi(f_i)$ (Kuzin et al., 2017).

2. Mathematical Assumptions and Structural Constraints

The LSS model enforces several statistical and combinatorial assumptions on data and signal structure:

• Uniform feature law: $X\sim\text{Uniform}([0,1]^p)$, which ensures streamlined probabilistic analysis and threshold coverage.
• Response boundedness: $|Y|<1$ for all $x$.
• Non-overlapping basic interactions: $$\forall j_1\neq j_2, S_{j_1}\cap S_{j_2} = \varnothing$$ regulates the complexity and guarantees identifiability of interacting features.
• Sparsity regime: The "signal support" $s=|\bigcup_j S_j|$ is fixed, and the regime $\log p / n \to 0$ characterizes asymptotic recovery guarantees as dimensionality and sample size scale (Vuk et al., 11 Dec 2025).

For the spike-and-slab GP hierarchy, spatial and temporal dependency structures are encoded via covariance kernels $K$ parameterized by length-scales $\ell_s$, $\ell_t$ and marginal variance $\sigma_f^2$, often subject to weakly informative hyperpriors (e.g., inverse-Gamma for variance, log-normal for length-scale) (Kuzin et al., 2017).

3. Local Support and Signed Feature Interactions

Local prediction for a test point $x^*$ is determined by which basic interactions $S_j$ are active—the subset of features or groups for which $x^*\in R_j$, with $R_j = \{x: x_k \le \gamma_k$ for all $k \in S_j\}$. This motivates the concept of Basic Signed Interaction (BSI):

• For each feature $k$, $(k, -1)$ represents "small" ($x_k \le \gamma_k$) and $(k, +1)$ "large" ($x_k > \gamma_k$).
• $S_j^-$ encodes $(k, -1): k\in S_j$ (all features meeting the threshold); $S_j^+$ is possible for $|S_j|=1$ and $x^*_k > \gamma_k$.
• The local support $$S_0(x^*) = \bigcup_{\{j: S_j^\pm \text{ is a BSI for } x^*\}} S_j$$ tracks features actively driving the model output at $x^*$ (Vuk et al., 11 Dec 2025).

In the Bayesian regression formulation, clusters of active weights emerge where the GP latent field $f$ is locally elevated, inducing spatial clusters of nonzeros through correlated spike probabilities $\pi_i=\Phi(f_i)$ (Kuzin et al., 2017).

4. Recovery Guarantees and Inference Algorithms

LSS recovery theorems establish provable consistency for both interpretable random forest methods and hierarchical Bayesian regression under model-specific conditions:

• For random forests, growing trees with $mtry\approx p$, no subsampling, and balanced splits (see conditions A1–A4), LocalLSSFind enumerates signed feature sets $S^\pm$ with high global depth-weighted prevalence and path prevalence. The central theorem asserts that, under proper threshold choices,

$$2^s b(\epsilon) < \pi_G < \frac{1}{2} C_m^s, \quad b(\epsilon) < \pi_L < 1$$

where $$b(\epsilon) = \left( \frac{4\epsilon}{C_\beta^2 C_\gamma^{2 \max_j |S_j| - 1}} \right)^{\tilde C}$$ and $\tilde C = C_m^{2s}/\ln(1/C_\gamma)$,

the method consistently recovers all BSIs for $x^*$ of size $\leq s_{max}$, and thus the full local support $S_0(x^*)$, with probability tending to 1 as $n\to\infty$ (Vuk et al., 11 Dec 2025).

• For spike-and-slab hierarchical regression, posterior inference is tractably achieved via Expectation Propagation (EP), which approximates the joint $p(w,z,f,\theta|y)$ by iteratively updating site-wise Gaussian and exponential-family factors, matching local moments under cavity and tilted distributions. EP, as specified, provides analytic Gaussian marginals and pointwise estimates $E[z_i]$ for sparse support recovery (Kuzin et al., 2017).

5. Model Instantiations and Special Cases

The LSS concept admits several practical instantiations:

• Piecewise-constant regression surfaces: Constructed by axis-aligned AND rules, where non-overlapping sets $S_j$ correspond to active combinations yielding a constant output increment $\beta_j$ in associated input regions.
• Interactions of bounded order: All $|S_j| \leq s_{max}$, restricting the complexity of signal structure.
• Simulation regime: Additive Gaussian noise $$Y = \sum_j \prod_{k\in S_j} \mathbf{1}(X_k < \tau) + N(0,\sigma^2)$$ can be incorporated, but theoretical results target noise-free or bounded response settings (Vuk et al., 11 Dec 2025).
• Spatio-temporal regression: In the Bayesian GP setup, local activation clusters can appear, move, and dissipate in space-time—enabled by top-level GP evolution of latent means (Kuzin et al., 2017).

A plausible implication is that LSS can be flexibly adapted to situations with heterogeneous local signal structure, especially where interpretability in terms of explicit logical or probabilistic rules is paramount.

6. Interpretability and Feature Directionality

Interpretability within the LSS regime is enhanced by the explicit sign information attached to recovered interactions:

• Each signed interaction $S^\pm$ encodes not just which feature(s) are locally relevant, but also whether the specific local prediction is driven by their "small" ($x_k \le \gamma_k$) or "large" ($x_k > \gamma_k$) values.
• This directionality allows feature- and group-level attribution for individual predictions, supporting nuanced feature importance analysis at local (per-sample) resolution.
• In regression and random forest settings, such decomposition serves to refine the explanatory power beyond global variable importance measures, connecting model output directly to interpretable, axis-aligned rules (Vuk et al., 11 Dec 2025).

In hierarchical Bayesian implementations, analytic marginals and local spike probabilities further enhance post hoc interpretation of recovered supports.

7. Computational Considerations and Extensions

The computational complexity of LSS model fitting depends on instantiation:

• Random forest-based local recovery scales with forest size and enumeration of signed feature sets; balanced tree construction and prevalence calculations are required for theoretical guarantees.
• Hierarchical spike-and-slab models with GP priors incur $O(N^3)$ complexity due to kernel inversion, generally alleviated via inducing point or Kronecker methods, bringing complexity towards $O(NM^2)$ for $M$ auxiliary variables (Kuzin et al., 2017).
• EP is favored for speed and analytic tractability compared to MCMC, especially in high-dimensional regimes. Point estimation via $E[z_i]$ and posterior marginalization are direct.

This suggests scalable practical implementations are available, supporting adoption of LSS methodologies in applications demanding structure-aware local sparsity, such as personalized medicine, genomics, and spatially resolved regression.

---

Key references:

• Vuk et al., "Provable Recovery of Locally Important Signed Features and Interactions from Random Forest" (Vuk et al., 11 Dec 2025).
• "Structured Sparse Modelling with Hierarchical GP" (Kuzin et al., 2017).