Restricted Boltzmann Machine Overview

• Restricted Boltzmann Machines are stochastic neural networks featuring a visible layer and a hidden layer with pairwise interactions but no intra-layer connections.
• They are applied to model complex spectral densities and construct data-adaptive kernels for advanced regression tasks.
• Integration with quantum annealing enables efficient sampling and gradient-based optimization to enhance kernel performance in real-world applications.

A Restricted Boltzmann Machine (RBM) is a stochastic neural network architecture consisting of a visible layer and a hidden layer with undirected, pairwise connections between them and no intra-layer connections. The RBM defines a joint Boltzmann distribution over binary-valued visible units $\mathbf{v} \in \{-1,+1\}^{N_v}$ and binary-valued hidden units $\mathbf{h} \in \{-1,+1\}^{N_h}$, parameterized by biases for the visible ($\mathbf{b}$) and hidden ($\mathbf{c}$) layers and a weight matrix $\mathbf{W}$.

1. Energy Function and Gibbs–Boltzmann Distribution

The RBM assigns an energy function

$$E(\mathbf{v}, \mathbf{h}) = -\sum_{i=1}^{N_v} b_i v_i - \sum_{j=1}^{N_h} c_j h_j - \sum_{i,j} W_{ij} v_i h_j$$

which defines the joint probability

$$P(\mathbf{v}, \mathbf{h} \mid \mathbf{b}, \mathbf{c}, \mathbf{W}) = \frac{\exp[-E(\mathbf{v}, \mathbf{h})]}{Z}$$

where $Z$ denotes the partition function over all visible and hidden configurations.

The absence of intra-layer connections (i.e., restricted architecture) renders the conditional distributions $P(\mathbf{h} \mid \mathbf{v})$ and $P(\mathbf{v} \mid \mathbf{h})$ fully factorized, enabling efficient block Gibbs sampling for stochastic learning and inference (Hasegawa et al., 13 Jan 2026).

2. Role in Spectral Density Modeling

In kernel learning frameworks, an RBM can be deployed to model the spectral density $p(\omega)$ within Bochner's theorem for shift-invariant, positive-definite kernels: $$k(\Delta \mathbf{x}) = \mathbb{E}_{\omega \sim p(\omega)}[\cos(\omega^{\top} \Delta \mathbf{x})]$$ where $\Delta \mathbf{x} = \mathbf{x} - \mathbf{x}'$. The RBM leverages its capacity to approximate complex, potentially multimodal distributions by learning $p(\omega)$ as an implicit spectral distribution over discrete binary vectors $\mathbf{v}$. These are later mapped to continuous frequencies $\omega$ through auxiliary conditional models (Hasegawa et al., 13 Jan 2026).

This approach supports the construction of data-adaptive kernels whose spectral properties transcend the limitations of fixed-form parametric distributions (e.g., a single Gaussian).

3. Quantum Annealing-Based Sampling and Frequency Mapping

RBM configurations $(\mathbf{v}, \mathbf{h})$ are sampled using quantum annealing (QA) hardware, e.g., the D-Wave Advantage_system4.1, by compiling the energy function into an Ising Hamiltonian. Quantum annealing yields samples that approximate the Gibbs–Boltzmann distribution of the RBM for the device's temperature and noise regime.

Each sampled vector $\mathbf{v}$ is transformed into a $d$-dimensional continuous frequency $\omega$ (matching the data dimension) via a Gaussian–Bernoulli conditional: $$P(\omega \mid \mathbf{v}, \mathbf{a}, \mathbf{U}, \boldsymbol{\sigma}) = \prod_{i=1}^d \mathcal{N}\left(\omega_i \,\middle|\, a_i + \sum_{j} U_{ij} v_j,\, \sigma_i^2 \right)$$ with learnable offsets $\mathbf{a}$, weights $\mathbf{U}$, and standard deviations $\boldsymbol{\sigma}$ ($\sigma_i^2 = \exp(z_i)$), ensuring a flexible mapping from binary RBM states to the continuous spectral domain (Hasegawa et al., 13 Jan 2026).

4. Integration with Random Fourier Features and Kernel Construction

Given $S$ frequency samples $\{\omega_s\}_{s=1}^S \sim p(\omega)$ constructed via the RBM-QA pipeline, the real-valued random Fourier feature (RFF) map

$$\varphi_\theta(\mathbf{x}) = \frac{1}{\sqrt{S}}\left[ \cos(\omega_1^{\top} \mathbf{x}),\, \ldots,\, \cos(\omega_S^{\top} \mathbf{x}),\, \sin(\omega_1^{\top} \mathbf{x}),\, \ldots,\, \sin(\omega_S^{\top} \mathbf{x}) \right]^\top$$

enables kernel approximation

$$k_\theta(\mathbf{x}, \mathbf{x}') \approx \varphi_\theta(\mathbf{x})^\top \varphi_\theta(\mathbf{x}')$$

with the simplified cosine-only average also used: $$k(\Delta \mathbf{x}) \approx \frac{1}{S} \sum_{s=1}^S \cos(\omega_s^\top \Delta \mathbf{x})$$

The resulting kernel is not a traditional fixed Gaussian kernel, but is data-adaptive as its spectral density is learned through the RBM and QA-driven sampling loop (Hasegawa et al., 13 Jan 2026).

5. Optimization within Kernel Regression Frameworks

RBM parameters, along with frequency mapping parameters, are trained end-to-end with respect to the leave-one-out Nadaraya–Watson (NW) mean squared error (MSE) objective under squared-kernel weights: $w_{ij} = k_{ij}^2$ to prevent denominator collapse and enhance the contrast of similar neighbors. The prediction for each training instance $i$ is computed as

$$\hat{y}_i^{(-i)} = \frac{\sum_{j\neq i} w_{ij} y_j}{\sum_{j\neq i} w_{ij} + \epsilon}$$

where $\epsilon \ll 1$ stabilizes the denominator numerically.

Gradients with respect to kernel and hence RBM parameters are propagated through score-function identities for Boltzmann models, utilizing: $$\frac{\partial \log p_\theta(\omega)}{\partial \theta} = -\frac{\partial E_\theta(\omega)}{\partial \theta} - \mathbb{E}_{p_\theta}\left[-\frac{\partial E_\theta(\omega)}{\partial \theta}\right]$$ and all parameter updates are handled via stochastic gradient steps (Hasegawa et al., 13 Jan 2026).

6. Performance Characteristics and Structural Insights

Empirical evaluation across standardized regression datasets demonstrates that the RBM-driven kernel learning framework (KLNW: Kernel Learning with Nadaraya–Watson) monotonically reduces training loss, induces structural modifications in the kernel matrix—specifically in off-diagonal entries reflecting learned neighbor relations—and systematically outperforms both fixed Gaussian kernel NW regression and Gaussian-kernel SVR in $R^2$ and RMSE.

Key performance results (test $R^2$ at $S=2000$ random features):

Dataset	KLNW $R^2$	Baseline NW $R^2$
bodyfat	0.910	0.715
mg	0.666	0.643
energy	0.980	0.932
ccs	0.748	0.677

The learned spectral histograms are strongly non-Gaussian and often multimodal, confirming that the RBM–QA procedure discovers data-adaptive, complex frequency structures. Increasing $S$ during inference consistently improves prediction accuracy due to reduced Monte Carlo variance in the RFF approximation (Hasegawa et al., 13 Jan 2026).

7. Application-Specific Adjustments and Inference Procedures

At inference, endpoint queries (test samples with any coordinate in the lowest or highest 1% of the empirical training marginal) invoke a local linear regression (LLR) correction utilizing the squared-kernel weights: $w_{*j} = k_\theta(\mathbf{x}_*, \mathbf{x}_j)^2$ A weighted least squares problem is solved to fit $$y \approx \beta_0 + \beta^\top (\mathbf{x} - \mathbf{x}_*)$$; the intercept $\hat{\beta}_0$ serves as the local linear prediction. For interior points, the NW prediction is used. The LLR correction further improves the metrics, including surpassing SVR in certain regimes.

A plausible implication of these findings is that the representational flexibility of an RBM, particularly when coupled with quantum annealing for efficient, hardware-based sampling, enhances adaptive kernel construction for regression beyond parametric spectral models and classical gradient-based estimators. The observed multimodal spectral densities learned by the RBM suggest its practical effectiveness in modeling structured frequency mixtures relevant for kernel-based machine learning (Hasegawa et al., 13 Jan 2026).