SIREN: Sinusoidal Representation Networks

Updated 8 February 2026

SIREN architectures are multilayer perceptrons that use sine activations to represent continuous signals, enabling accurate modeling of images, audio, and physical fields.
The design leverages principled weight initialization and frequency scaling to stabilize gradient propagation and preserve both low- and high-frequency details.
Variants like FM-SIREN and SIREN² enhance frequency diversity and adaptability, achieving improved PSNR, noise robustness, and efficient parameter utilization.

A SIREN (Sinusoidal Representation Network) is a multilayer perceptron (MLP) that uses the sinusoidal activation function in each hidden layer to parameterize continuous, differentiable implicit neural representations (INRs) of signals such as images, audio, 3D geometry, and physical fields. SIREN architectures are defined by their characteristic use of sine activation, principled initialization to maintain signal propagation and frequency fidelity, and the ability to directly model signal derivatives necessary for representing physical PDEs. Subsequent advances have introduced neuron-specific frequency multipliers (e.g., FM-SIREN), adaptive weight initialization strategies (WINNER/SIREN²), and specialized variants for diverse tasks such as noise-robust INR, high-fidelity audio compression, and multi-modal registration.

1. Mathematical Definition and Architectural Principles

A canonical SIREN consists of a fully connected stack of layers, where each hidden layer applies an affine transformation followed by the elementwise sine nonlinearity. For input $x \in \mathbb{R}^{d_{in}}$ , parameters $\theta = \{W_i, b_i\}_{i=0}^n$ , and layer input/output dimensions $M_i, N_i$ , the mapping is:

$\varphi_\theta(x) = W_n(\varphi_{n-1} \circ \ldots \circ \varphi_0)(x) + b_n,$

where each hidden layer applies

$\varphi_i(u) = \sin(W_i u + b_i).$

In practice, SIREN uses a frequency scaling hyperparameter $\omega_0$ for the first layer only:

$\varphi_0(x) = \sin(\omega_0 W_0 x + b_0).$

In all subsequent layers, $\omega_0 = 1$ (unless using alternative variants). This configuration ensures that the network’s post-activation distribution remains stable across layers and retains sensitivity to both low- and high-frequency signals (Sitzmann et al., 2020).

Weight initialization is critical for signal propagation: for layers $i = 1, \ldots, n-1$ , set $W_i \sim \text{Uniform}(-\sqrt{6/M_i}, +\sqrt{6/M_i})$ ; for the first layer, $W_0 \sim \text{Uniform}(-\sqrt{6/M_0}/\omega_0, +\sqrt{6/M_0}/\omega_0)$ . This stabilizes the variance of pre-activations and ensures sinusoids do not saturate or become degenerate, preserving the network’s representational bandwidth (Sitzmann et al., 2020).

2. Theoretical Rationale and Frequency Properties

The use of periodic activation is motivated by the superior capacity of $\sin(\cdot)$ to represent signals with rich local and global oscillatory content. Unlike ReLU or tanh nonlinearities that suppress or rapidly decay high-frequency components, the sine function admits nonzero derivatives of all orders and can represent arbitrarily oscillatory features.

Deeper SIRENs are theoretically analogous to deep Fourier synthesizers; stacking sine activations constructs increasingly complex superpositions of harmonics. Maintaining pre-activation variance at initialization (via principled weight scaling) is essential to preserve high-frequency expressivity and avoid frequency collapse through depth (Sitzmann et al., 2020).

Closed-form derivatives (from automatic differentiation) make SIREN architectures directly suitable for learning solutions and derivatives of PDEs. For example, for

$\varphi_i(u) = \sin(W_i u + b_i), \quad \frac{\partial \varphi_i}{\partial u} = \cos(W_i u + b_i) \cdot W_i,$

higher-order derivatives follow recursively. This analytic differentiability underpins applications from signal fitting to PDE-solving (Sitzmann et al., 2020).

3. Variants: Frequency-Multiplied and Adaptive Initialization SIRENs

FM-SIREN

FM-SIREN addresses the core limitation of original SIRENs: feature redundancy induced by a layer-wide fixed frequency multiplier $\omega_0$ , leading to correlated, overlapping frequency responses across neurons. Drawing from Nyquist-Shannon sampling theory and the discrete sine transform (DST), FM-SIREN assigns to each neuron in a layer a neuron-specific multiplier,

$\omega_k = \frac{k}{M} f_{Nyquist}, \quad k = 0, \ldots, M-1, \quad f_{Nyquist} = \frac{f_s}{2},$

where $M$ is the number of neurons and $f_s$ the domain sampling rate. This set $\{\omega_k\}$ spans $[0, f_{Nyquist}]$ , eliminating redundancy and ensuring near-orthogonality across features (Alsakabi et al., 27 Sep 2025).

Empirically, this yields a 49–50% reduction in off-diagonal covariance of hidden features (quantified via the Frobenius norm of covariance matrices), outperforms baseline SIREN in PSNR and MSE across audio, image, 3D, and NeRF fitting, and does so at unchanged computational cost (Alsakabi et al., 27 Sep 2025). FM-SIREN matches or exceeds the performance of a deep (5-layer) SIREN with only 2 layers.

WINNER (SIREN²)

SIREN exhibits spectral bias at initialization: pre-activations have low-frequency spectral support, impeding learning of signals with significant high-frequency content. When the target signal's spectral centroid lies outside the network's support, a "spectral bottleneck" arises: the network collapses to near-zero outputs for all frequencies.

WINNER (Weight Initialization with Noise for Neural Representations; SIREN²) injects adaptive Gaussian noise into the first two layers’ weights, with noise magnitude determined by the spectral centroid $\psi$ of the target signal ( $\psi=2\sum_k k|\hat{y}(k)|/\sum_k |\hat{y}(k)|$ ). This increases pre-activation variance and widens spectral support without increasing trainable parameters:

$W^{(l)}_{jk} \leftarrow U(-c, c) + \eta_{jk}^{(l)},\quad \eta_{jk}^{(l)} \sim \mathcal{N}\left(0, \frac{s_l}{\omega_0}\right),$

with $s_0, s_1$ (noise scales) computed as functions of $\psi$ .

Empirically, SIREN² achieves state-of-the-art PSNR in audio, image, and 3D fitting, outperforming both SIREN and parameter-intensive SIREN+RFF, and also enhances denoising in DIP/Noise2Self setups. The analytic effect on spectral energy is accurately predicted by $\mathrm{Var} \approx 1 + (d \cdot s^2)/2$ (Chandravamsi et al., 16 Sep 2025).

4. Key Applications and Performance

SIREN and its variants serve as implicit neural representations for a range of tasks:

Signal Fitting: SIRENs and FM-SIRENs precisely reconstruct 1D audio, 2D images, and 3D shapes from coordinate–value pairs, achieving high PSNR and low MSE compared to non-periodic or PE-based MLPs (Sitzmann et al., 2020, Alsakabi et al., 27 Sep 2025).
Physical PDE Solving: SIRENs are directly applied to systems defined via derivatives, such as:
- Poisson equation: via supervision only on gradients or Laplacians.
- Eikonal equation: for signed distance function (SDF) learning with norm and normal constraints.
- Helmholtz/wave equation: for learning complex wavefields and FWI velocity models (Sitzmann et al., 2020).
Noise-Robust INR: Appropriately limiting width, depth, or frequency scaling enables SIREN to function as a mesh-free, noise-robust regressor for pressure fields from image velocimetry in challenging, noisy environments—outperforming classical mesh-based alternatives (Miotto et al., 29 Jan 2025).
Audio Compression: Architecture variants (Siamese SIREN) share backbones but separate output heads, facilitating implicit audio compression and self-noise estimation with substantially reduced parameter counts and improved objective and perceptual metrics after quantization (Lanzendörfer et al., 2023).
Registration and Multi-modal Fusion: SIREN architectures have been extended to semantic-guided registration pipelines for multi-robot Gaussian Splatting maps and as collaborative transformer systems for text-to-audio generation (Shorinwa et al., 10 Feb 2025, Wang et al., 6 Oct 2025).

5. Training, Initialization, and Implementation Details

SIRENs require careful architectural configuration:

Depth/Width: Depths of 5–10 layers and hidden widths of 256–512 neurons balance representational capacity and runtime (Sitzmann et al., 2020).
Initialization: Principled uniform initialization (as outlined above) is crucial to avoid vanishing/exploding gradients and to maintain frequency propagation.
Optimization: Training is via Adam (default $\beta_1, \beta_2, \varepsilon$ ), with learning rates in $10^{-4}$ – $10^{-3}$ .
Loss Functions: For regression, use MSE on value or derivatives; for PDE constraints, additional loss terms target gradients, Laplacians, or physical constraints.
Gradient Supervision: For PDE tasks, losses on spatial derivatives or Laplacians are computed via autograd.
Sampling: Coordinate minibatches are directly sampled from the input domain (e.g., spatial, spatiotemporal grids).

The frequency multipliers in FM-SIREN are computed once per layer (e.g., per audio sampling rate, image dimension, or voxel grid size) and stored as an $\mathcal{O}(M)$ -sized vector per layer (Alsakabi et al., 27 Sep 2025).

6. Comparative Empirical Results

Key empirical results for SIREN and variants include:

Architecture	Audio PSNR	Image PSNR	3D IoU/PSNR	NeRF PSNR	Notes
SIREN	13.4–59.4	21.3–38.9	0.960	33.02	Baseline, fixed $\omega_0$
FM-SIREN	62.7	32.29	0.990	33.17	10–50% reduction in redundancy, faster
SIREN² (WINNER)	62.7–95.2	36.1–75.2	55 dB	–	Adaptive freq. support, state of art
Siamese SIREN	–	–	–	–	40% fewer params, best audio metrics

SIREN²/WINNER matches or exceeds parameter-heavy SIREN+RFF, without increased optimizer or memory cost (Chandravamsi et al., 16 Sep 2025). FM-SIREN matches deep classic SIRENs with only 2 layers, and in NeRF reconstruction yields 14% faster training at unchanged fidelity (Alsakabi et al., 27 Sep 2025).

7. Limitations and Open Directions

Spectral Bottleneck: Even with appropriate initialization and frequency allocation, SIREN’s ability to learn highest frequencies is fundamentally linked to architecture and initialization. WINNER can underfit extremely low-frequency content if noise scales are excessive (Chandravamsi et al., 16 Sep 2025).
Parameter Sensitivity: FM-SIREN’s frequency allocation requires knowledge of the signal domain’s effective sampling rate. Incorrect estimation can degrade near-orthogonality of hidden features (Alsakabi et al., 27 Sep 2025).
Hyperparameter Tuning: Frequency multipliers in vanilla SIREN ( $\omega_0$ ), initialization constants, or adaptive noise scales ( $s_0, s_1$ ) are task/domain-specific. Porting to new signal domains may require retuning.
Generalization to Other Periodic Activations: Research suggests extending adaptive initialization and neuron-specific frequency selection to broader classes of periodic activations and positional encoding schemes is a promising avenue (Chandravamsi et al., 16 Sep 2025).
Integration with Hypernetworks: SIREN functions parameterized by hypernetworks support priors over distribution families of signals but inherit any representational limitations of the core SIREN.

SIREN architectures and their recent extensions define a mathematically principled and empirically validated toolkit for high-fidelity implicit neural representation, with both theoretical underpinnings grounded in harmonic analysis and practical advances in frequency diversity, initialization, compression, and robustness (Sitzmann et al., 2020, Alsakabi et al., 27 Sep 2025, Chandravamsi et al., 16 Sep 2025, Miotto et al., 29 Jan 2025, Lanzendörfer et al., 2023).