Meta-Continual Learning of Neural Fields

Abstract: Neural Fields (NF) have gained prominence as a versatile framework for complex data representation. This work unveils a new problem setting termed \emph{Meta-Continual Learning of Neural Fields} (MCL-NF) and introduces a novel strategy that employs a modular architecture combined with optimization-based meta-learning. Focused on overcoming the limitations of existing methods for continual learning of neural fields, such as catastrophic forgetting and slow convergence, our strategy achieves high-quality reconstruction with significantly improved learning speed. We further introduce Fisher Information Maximization loss for neural radiance fields (FIM-NeRF), which maximizes information gains at the sample level to enhance learning generalization, with proved convergence guarantee and generalization bound. We perform extensive evaluations across image, audio, video reconstruction, and view synthesis tasks on six diverse datasets, demonstrating our method's superiority in reconstruction quality and speed over existing MCL and CL-NF approaches. Notably, our approach attains rapid adaptation of neural fields for city-scale NeRF rendering with reduced parameter requirement.

• The paper presents MCL-NF, a novel framework that mitigates catastrophic forgetting using a modular architecture with shared meta-learned initialization.
• It employs MAML for few-step task adaptation and integrates FIM loss to prioritize high-impact samples, enhancing reconstruction quality.
• Experiments on image, video, audio, and NeRF tasks show that MCL-NF outperforms traditional methods in speed, reconstruction metrics, and memory efficiency.

This paper introduces a novel problem setting called Meta-Continual Learning of Neural Fields (MCL-NF), which aims to train Neural Fields (NFs) that can learn sequentially (continually) from data streams while also adapting rapidly to new data (meta-learning). The goal is to overcome limitations like catastrophic forgetting and slow convergence often seen in traditional NF training and existing Continual Learning of Neural Fields (CL-NF) methods.

The proposed solution combines a modular architecture with optimization-based meta-learning (specifically MAML).

• Modularization: Inspired by large-scale NeRF models (e.g., MegaNeRF, SwitchNeRF), the approach assigns different modules (small MLPs) to different tasks or spatial/temporal segments of the data. This inherently mitigates catastrophic forgetting by isolating task-specific parameters. A key aspect is the use of shared initialization for all modules, derived through meta-learning. This allows modules to adapt quickly and enables flexibility in dynamically adding new modules as needed, unlike fixed modular structures.
• Meta-Learning (MAML): MAML is used to learn a good shared initialization ($\theta_{\mathrm{shared}}$) for the modules. This initialization allows for rapid adaptation (few-step fine-tuning) when encountering new tasks or data within a continual learning sequence.
• Fisher Information Maximization (FIM) Loss: To further enhance learning efficiency and generalization, especially when tasks overlap (e.g., similar structures in different city blocks), the paper introduces the FIM loss. This loss function re-weights training samples based on their Fisher Information contribution. It computes a weight $$w(\theta_i) = 1 + \lambda \cdot \mathrm{tr}(\mathbf{g}(\theta_i)^T \mathbf{F}_i^{-1} \mathbf{g}(\theta_i))$$, where $\mathbf{g}$ is the score function (gradient of the log-likelihood) and $\mathbf{F}$ is the Fisher Information Matrix. This weighting prioritizes samples that cause significant changes to important model parameters, effectively focusing learning on the most informative data points. This sample-level weighting is contrasted with parameter-level regularization methods like EWC. The paper provides theoretical justification, linking FIM loss to KL-divergence approximation and mutual information maximization, along with proofs for convergence guarantee and generalization bounds for FIM-based SGD.

The overall algorithm involves:

1. Meta-training to find a shared initialization $\theta_{\mathrm{shared}}$.
2. During continual learning (meta-test time), initializing new modules with $\theta_{\mathrm{shared}}$.
3. Inner-loop updates for task-specific adaptation using the FIM loss.
4. Outer-loop updates (optional during meta-test, primary during meta-training) to refine the initialization based on performance across tasks.

Experiments were conducted on diverse tasks and datasets:

• Image Reconstruction: CelebA, FFHQ, ImageNette (splitting images into patches as sequential tasks).
• Video Reconstruction: VoxCeleb2 (treating frames sequentially).
• Audio Reconstruction: LibriSpeech (processing audio segments).
• View Synthesis (NeRF): MatrixCity (large-scale city rendering, splitting scenes into grids).

Results demonstrate that the proposed method (both with and without FIM loss, denoted 'Ours (MIM)' and 'Ours (mod)' respectively) consistently outperforms baseline MCL methods (OML, MAML+CL) and CL methods (EWC, ER, MER) in terms of reconstruction quality (PSNR, SSIM, PESQ) and learning speed (faster convergence during adaptation steps). Notably, it achieves strong performance on city-scale NeRF rendering with fewer parameters compared to standard single-network NeRFs, showing good scalability and efficiency. The FIM loss variant ('Ours (MIM)') often provides a slight edge, particularly in stability and final quality. Computational analysis highlights the method's memory efficiency compared to baselines.

The authors conclude that MCL-NF, via modularization and meta-learning with FIM loss, offers an effective strategy for rapid and continual learning of neural fields, particularly beneficial for applications involving dynamic data streams and resource constraints (e.g., drones, edge devices). Limitations include potential degradation over very long/dissimilar task sequences and increased memory with a very large number of tasks.