Implicit Neural Representation Hyperpriors

• Implicit Neural Representation-Based Hyperpriors are mechanisms that encode structured, cross-signal inductive biases over INR parameters using meta-learned initializations or semantic priors.
• They leverage techniques like meta-learning with sparsity masks and semantic conditioning through pretrained networks to generate efficient, adaptive INR weights.
• Empirical results using Meta-SparseINR and SPW show improved PSNR, enhanced weight diversity, and significant parameter savings compared to traditional dense representations.

Implicit neural representation (INR)-based hyperpriors are mechanisms that encode structured, cross-signal information as inductive biases over INR parameters, enabling more memory- and adaptation-efficient signal representations. These methods leverage either meta-learned initialization or explicit conditioning on semantic priors, acting as shared or signal-specific hyperparameters that shape the subsequent optimization or instantiation of INR models. Two representative approaches in this space are meta-learned sparse hyperpriors (Lee et al., 2021) and semantic-prior-induced INR weight generation (Cai et al., 2024).

1. Mathematical Framework for INR-Based Hyperpriors

Implicit neural representations map continuous coordinates in a signal's domain to values in its codomain, with the mapping performed by a neural network $f(x;\theta)$ parameterized by weights $\theta \in \mathbb{R}^d$. The hyperprior perspective treats the process of selecting or adapting $\theta$ (for each new signal or task) as drawing from a distribution governed by hyperparameters or meta-knowledge that was learned across a population of signals.

In meta-learning sparse INRs, the hyperprior is a meta-learned initialization $\theta^{(0)}$, subject to a sparsity-inducing mask $M \in \{0,1\}^d$. The objective is: $$\min_{\,\theta^{(0)} \in \mathbb{R}^d}\;\min_{M \in \{0,1\}^d,\, \|M\|_0 \leq \kappa} \;\frac{1}{N} \sum_{j=1}^N L_j (M \odot \theta_j^{(t)}(\theta^{(0)}))$$ where $\theta_j^{(t)}$ is the parameter after $t$ steps of task-specific adaptation on task $j$.

In SPW (Semantic Priors in Weights), hyperpriors are injected explicitly via semantic code $z$ derived from a pretrained feature extractor, leading deterministically (through a weight synthesis MLP) to INR weights $W$: $z = \mathrm{SNN}(X), \quad W = f_{\phi}(z)$ where $X$ is the raw signal, $\mathrm{SNN}$ is a fixed feature network, and $f_{\phi}$ is a trainable weight generator MLP (Cai et al., 2024). This is formalized as a joint Dirac model $$p(W, z | X) = \delta(z - \mathrm{SNN}(X)) \delta(W - f_{\phi}(z))$$.

2. Meta-Learning and Sparsity as Hyperprior Construction

In meta-learned sparse INRs (Lee et al., 2021), hyperpriors are shaped through meta-learning with an explicit pruning procedure. The protocol involves the following:

• Inner loop: For each signal $T_j$, adapt $\theta^{(0)}$ with the current mask $M$ for $t$ gradient steps on loss $L_j$.
• Outer loop: Update $\theta^{(0)}$ via meta-gradient using post-adaptation weights and aggregate losses over all tasks.
• After each outer iteration, enforce a global sparsity constraint by retaining only the highest-magnitude weights (global magnitude-based pruning), updating $M$ accordingly.

No $\ell_0$ or $\ell_1$ penalty is introduced; sparsity is imposed structurally by maintaining and updating $M$. After each pruning, additional meta-learning steps fine-tune $\theta^{(0)}$ for stability on the pruned structure.

The architectural instantiations include SIREN (sinusoidal-MLP) and a Fourier Feature Network (FFN) variant, with sparsity masks applied consistently over all linear weights without altering the network structure.

3. Semantic-Prior Hyperpriorization and INR Weight Generation

SPW introduces semantic hyperpriors by reparameterizing INR weights via a deterministic two-stage pipeline (Cai et al., 2024):

1. Semantic Neural Network (SNN): A frozen EfficientNet-B7 pretrained on ImageNet, taking the target signal $X$ and producing multi-scale global semantic descriptors by global average pooling over all stages and concatenating the results.
2. Weight Generation Network (WGN): For each INR layer, a three-layer MLP ("inverted bottleneck" architecture) maps the semantic vector $v_s$ to layer weight tensors. Expansion factors and MLP depth are ablated; three layers with moderate expansion optimize PSNR performance.

The SPW pipeline operates deterministically—no explicit probabilistic prior is imposed. During training, only the WGN weights are updated. After training, SNN and WGN are discarded; only the generated INR weights are kept, so inference is as efficient as in standard INRs.

4. Optimization Procedures

Meta-Learning Sparse INRs:

• Initialization: Standard SIREN weights, full (all-ones) mask $M$.
• Phase 1: Dense MAML pre-training.
• Phase 2: Iteratively prune a fixed fraction $\gamma$ of smallest-magnitude weights, re-meta-training after each prune until the target parameter count $\kappa$ is reached.
• Test-time: For a new signal, adapt the pruned and meta-initialized weights for $t$ steps with mask $M$.

SPW Weight Synthesis:

• Compute semantic vector $v_s$ for input $X$ once.
• Run each WGN to generate layerwise INR weights.
• Form the final INR for the signal, using only the generated weights for inference.
• Adam is used for optimization, with task-dependent settings.

Gradient updates flow only through the WGN. The semantic feature extractor SNN remains fixed throughout. The forward INR is constructed solely from generated weights for each new input.

5. Empirical Results and Comparative Analyses

Meta-SparseINR (Lee et al., 2021):

• Datasets: CelebA, Imagenette, 2D shape SDFs.
• Metric: PSNR after 100 per-signal Adam steps.
• Results (CelebA, SIREN, $\approx$8.7k params): Meta-SparseINR achieves 27.71 dB PSNR, outperforming Random-Prune (26.25 dB with 17k params) and matching Dense-Narrow (27.68 dB with 17.7k params) despite using $\leq 50\%$ of their parameters.
• Adaptation speed: With 2 inner steps, yields 24.6 dB (vs 21.5 dB for dense-narrow).
• Robustness is retained at high sparsity levels (down to 16% parameters).

SPW (Cai et al., 2024):

• Tasks: Image fitting (Kodak), CT, MRI, NeRF synthesis.
• Gains: SPW introduces systematic PSNR increases ($\sim$1 dB avg) over each baseline INR backbone.
• Rate–distortion: SPW sits $\sim$1 dB above non-SPW variants at all bitrates.
• Weight analysis: Lower KL-divergence self-similarity, higher entropy in weights, and greater first-layer activation diversity with SPW-injected weights.
• Ablations confirm both low- and high-level semantics contribute to optimal performance (full EfficientNet stages 1–7).
• WGN hyperparameter ablations reveal that three-layer, moderately expanded MLPs perform best.

Model	2D Img	2D CT	3D MRI	5D NeRF
SIREN	25.52	28.30	26.04	25.44
SPW-SIREN	26.61	29.14	26.82	25.86
PE-MLP	23.16	28.11	30.17	30.99
SPW-PE-MLP	24.06	29.25	30.99	31.52
MFN	25.25	27.97	27.24	31.04
SPW-MFN	26.13	28.92	27.71	31.47
WIRE	25.05	28.26	25.31	25.76
SPW-WIRE	25.74	28.96	25.94	26.15

A plausible implication is that encoding semantic priors as hyperprior information into INR weights improves representational capacity and reduces redundancy compared to purely optimization-based initializations.

6. Limitations and Extensions

• Meta-SparseINR’s reliance on iterative magnitude pruning and meta-learning can incur additional computation during training, though it yields parameter savings and rapid adaptation at test time (Lee et al., 2021).
• SPW’s dependency on a frozen large-scale SNN (EfficientNet-B7) restricts flexibility and increases the training footprint; alternatives such as lightweight or equivariant backbones (ViT, ConvNeXt) could reduce overhead (Cai et al., 2024).
• Both methods utilize deterministic hyperpriors—a learned probabilistic posterior (e.g., using normalizing flows for $q(z|X)$) would allow end-to-end training for tasks requiring probabilistic inference or compression.
• SPW’s approach can, in principle, be extended to dynamic or conditional INR settings (e.g., D-NeRF, video) by varying the semantic vector per timestep or condition, allowing instance- or context-adaptive parameterizations.

7. Significance and Outlook

Implicit neural representation-based hyperpriors unify learned inductive biases and explicit side information for parameter-efficient, high-quality signal representation and adaptation. By encoding cross-signal knowledge either in meta-learned initializations and structural masks (as in Meta-SparseINR (Lee et al., 2021)) or in extracted semantic codes that directly generate INR weights (as in SPW (Cai et al., 2024)), these methods offer new paradigms for balancing model compression, fast adaptation, and semantic expressivity in continuous function representations. This suggests that future work will likely further explore learned, flexible hyperprior distributions and enhanced semantic conditioning modalities to generalize INR-based modeling across broad application domains.