Masked Frequency Modeling (MFM)

• Masked Frequency Modeling is a self-supervised method where models infer missing frequency components after transformations like DFT or DCT to capture both global and local details.
• It employs diverse masking strategies—such as low-pass, high-pass, random, or learnable masks—with reconstruction objectives to improve robustness and feature learning.
• Empirical results demonstrate enhanced performance in image classification, time series forecasting, and hyperspectral data analysis by integrating frequency-based and spatial losses.

Masked Frequency Modeling (MFM) refers to a class of self-supervised or auxiliary learning strategies in which models are trained to infer missing or corrupted frequency components of data—typically after an explicit transformation into the frequency domain. By masking portions of the frequency spectrum and enforcing a reconstruction (or prediction) objective, MFM aims to enhance signal understanding, improve robustness, and foster superior representations across modalities, including images, time series, biosignals, hyperspectral data, and text.

1. Foundational Concepts and Motivation

Masked Frequency Modeling operates by selectively obscuring components in a transformed frequency domain, most commonly after a Discrete Fourier Transform (DFT) or Discrete Cosine Transform (DCT). This is in contrast to the more ubiquitous masked (spatial) modeling, in which directly masking input elements or patches serves as the pretext task. MFM seeks to address the heavy redundancy in spatial representations and to force models to learn both global statistical regularities (low frequencies) and local structural details (high frequencies). Empirical evidence across domains demonstrates that frequency-domain masking encourages the inference of both semantic content and fine-grained structure, enhancing generalization and robustness to distributional shift and adversarial noise (Xie et al., 2022, Liu et al., 2023, Gui et al., 2022, Mohamed et al., 6 May 2025, Fu et al., 2024).

2. Mathematical Formalism and Core Methodology

The canonical MFM pipeline encompasses the following steps:

1. Frequency Transformation: For an input $x$ (e.g., image, sequence), a DFT or DCT is applied. For images $x \in \mathbb{R}^{H \times W \times C}$, the DFT is computed independently per channel:

$$F(u,v) = \sum_{h=0}^{H-1}\sum_{w=0}^{W-1} x(h,w)\,e^{-i2\pi(uh/H + vw/W)}$$

1. Masking in Frequency Space: A binary or soft mask $M$ is applied to the transformed data. Mask designs include low-pass, high-pass, random, or learnable masks, with selection strategies tailored to the application (e.g., random circular masks for images (Xie et al., 2022), segment-wise for time series (Ma et al., 2024), patch-masking for DPMs (Iqbal et al., 2023)). In text modeling, mask probability is informed by token rarity (Kosmopoulou et al., 5 Sep 2025).
2. Reconstruction or Inference Objective:
   • For image or signal data, the masked frequency representation is inverse-transformed to obtain a corrupted version in the original domain, which is passed through an encoder-decoder pipeline. The decoder is tasked to predict the masked spectrum or reconstruct the uncorrupted input.
   • The typical loss is $\ell_1$ or $\ell_2$ over the masked spectrum or Fourier coefficients:

   $$\mathcal{L} = \mathbb{E}_{x,M} \left[ \|g((1-M) \odot F(x)) - (M \odot F(x))\|^p \right]$$

   where $g(\cdot)$ denotes the model, and $p$ is often set to $1$.

- In non-contrastive time series models, dual-branch architectures infer either the embedding of a masked-frequency variant from the unmasked signal (and vice versa) (Fu et al., 2024), minimizing the mean squared error between true and inferred embeddings.

1. Integration with Downstream or Dual-Domain Losses: In some settings, the total loss combines objectives in both frequency and spatial domains, especially in dual-domain masked image models for hyperspectral data (Mohamed et al., 6 May 2025):

$$\mathcal{L}_{\mathrm{total}} = \lambda_s \mathcal{L}_{\mathrm{spat}} + \lambda_f \mathcal{L}_{\mathrm{freq}}$$

with typical choice $\lambda_s = \lambda_f = 1$.

3. Architectural Variants and Domain-Specific Implementations

MFM has been instantiated in a diverse set of architectures and domains:

• Vision Transformers (ViT) and CNNs: Frequency-masked images can be processed by both transformer and convolutional backbones; MFM is agnostic to architecture. For ViT, masked images are patch-embedded as normal (Xie et al., 2022).
• Masked Diffusion Probabilistic Models: In masked-DDPM (mDDPM), frequency patch-masking augments the regular diffusion denoising schedule, compelling the network to reconstruct plausible spectra for healthy brain MRIs (Iqbal et al., 2023).
• Frequency-Attention Fusion: FAMT fuses DCT-based frequency importance with ViT self-attention scores to compose a joint masking/patch-throw schedule, reducing computational burden while emphasizing semantically critical content (Gui et al., 2022).
• Time Series and Biosignals: Patch-level DFT or DCT is leveraged, with adaptive masking at multiple temporal scales (e.g., MMFNet for multivariate time series forecasting uses learnable masks at fine, intermediate, and coarse frequency scales (Ma et al., 2024)). Frequency-aware transformers incorporate global Fourier filters to mix patches across time (Liu et al., 2023).
• Hyperspectral Data: SFMIM masks spectral vector components in the DFT domain, alternately applying low- and high-pass masks to maximize spectral-spatial correlation modeling (Mohamed et al., 6 May 2025).
• Language Modeling: Diffusion-based masked LLMs exploit frequency-informed masking at the token level, prioritizing rare tokens using global frequency statistics (Kosmopoulou et al., 5 Sep 2025).
• Non-Contrastive Representation Learning: Dual-branch prompt-based inference with random frequency masking yields continuous semantic embeddings, outperforming contrastive baselines in time series generalization (Fu et al., 2024).

4. Empirical Benefits and Experimental Findings

MFM consistently yields strong empirical outcomes in several regimes:

• ImageNet-1K: MFM matches or slightly exceeds patch-masked MAE and SimMIM in top-1 accuracy (ViT-B/16: MFM 83.1%, MAE 82.9%) and semantic segmentation (ADE20K mIoU: MFM 48.6%, MAE 48.1%) (Xie et al., 2022).
• Robustness: Models trained with MFM exhibit enhanced adversarial resistance (PGD accuracy: MFM 24.4%, MAE 11.2%) and lower corruption error rates (Xie et al., 2022).
• Hyperspectral Data: Frequency-only masking achieves OA 85.03%; dual-domain (frequency + spatial) masking reaches OA 91.15% on Houston 2013, confirming the complementarity of spectral regularization (Mohamed et al., 6 May 2025).
• Time Series Forecasting: Learnable frequency masks in MMFNet improve long-range MSE up to 6% over prior state-of-the-art, with ablation studies associating a 3.5% MSE drop directly to the mask module (Ma et al., 2024).
• Biosignal Pretraining: Frequency-masked autoencoders offer $\uparrow$5.5% accuracy gain, with notable robustness to modality dropout and substitution (Liu et al., 2023).

Empirical ablations reveal:

• Circular low/high band masks in vision outmatch square/rhombus shapes (Xie et al., 2022).
• 50/50 low/high frequency sampling strikes the optimal trade-off for vision and time series.
• Frequency-only masking provides a performance increase over random masking, but frequency-attention fusion (α≈0.5) achieves the highest gains (Gui et al., 2022).
• In text, frequency-informed masking of rare tokens leads to improvements on challenging linguistic tasks (Kosmopoulou et al., 5 Sep 2025).

5. Methodological Extensions and Design Variations

Significant design choices and extensions in MFM include:

• Mask Sampling: Methods span random, rank-based, or learnable masking, and may apply to contiguous bands, patches, or tokens. For example, MMFNet’s masks are learned via DCT-projected fragments at multiple scales (Ma et al., 2024), while FAMT combines soft frequency and attention weights for patch selection and discarding (Gui et al., 2022).
• Loss Coupling: Losses can be frequency-only, coupled frequency-spatial, or embedded in dual-branch inference (as in FEI (Fu et al., 2024)).
• Adaptive Masking: Adaptive or learnable masks enable dynamic focus depending on scale, modality, or signal properties.
• Multi-Scale Decomposition: Segmenting signals at multiple resolutions increases temporal sensitivity (as in MMFNet for LTSF (Ma et al., 2024)).
• Prompt-Based Inference: Prompting architectures—where masking itself becomes the “prompt” for another prediction branch—produce semantic embeddings with improved continuity (Fu et al., 2024).
• Frequency-Aware Attention: Some architectures introduce explicit frequency-mixing steps in attention or transformer blocks, enhancing cross-token communication with global spectral context (Liu et al., 2023).

6. Domain-Specific Applications

MFM has been effectively deployed in:

• Self-supervised visual pre-training for large-scale classification, segmentation, and adversarial robustness (Xie et al., 2022).
• Unsupervised anomaly detection in medical imaging, especially MRI, exploiting the regularity of frequency spectra in healthy anatomy (Iqbal et al., 2023).
• Hyperspectral image classification, where spectral masking complements spatial masking for fine material discrimination (Mohamed et al., 6 May 2025).
• Multivariate time series and long-horizon forecasting, leveraging scale-specific masks to capture both transients and trends (Ma et al., 2024).
• Non-contrastive self-supervised time series representation, with prompt-based frequency masking outperforming traditional contrastive methods on diverse datasets (Fu et al., 2024).
• Multimodal biosignal pretraining, enhancing representation invariance to channel composition and length (Liu et al., 2023).
• Data-efficient language modeling, where frequency-informed (rarity-weighted) masking in diffusion-based setups reduces sample complexity and improves the learning of rare words (Kosmopoulou et al., 5 Sep 2025).
• Efficient masked pretraining protocols, where frequency-guided patch “throwing” rapidly accelerates convergence and reduces resource demands (Gui et al., 2022).

7. Limitations, Challenges, and Research Directions

While MFM provides a powerful and flexible self-supervision mechanism, open challenges include:

• Selection of Masking Parameters: Mask ratio, mask shapes, and adaptive weighting require careful cross-validation; hyperparameters such as DCT bin cutoffs and scale fusion weights can influence performance (Gui et al., 2022).
• Spectral Artifacts: Naive masking in the frequency domain can introduce signal artifacts upon inversion, particularly when non-bandlimited signals are involved.
• Interpretability of Learned Spectra: While reconstructed spectra are physically interpretable in certain domains (e.g., biomedical), understanding how networks internalize spectral cues remains under-explored.
• Extending to Non-Euclidean Data: Most current approaches focus on grid-structured domains; extending MFM to graphs or manifolds requires new formulations.
• Fusion with Other Pretext Tasks: The synergistic effect of MFM with denoising, inpainting, and multimodal masking objectives is an active area of research.

A plausible implication is that MFM’s architectural agnosticism and generality will facilitate further cross-pollination between domains with inherent frequency structure, such as audio, geospatial time series, and molecular spectra, including multimodal and generative modeling applications (Liu et al., 2023, Gui et al., 2022, Fu et al., 2024, Kosmopoulou et al., 5 Sep 2025).