Visual State Space Block in Vision Models

• Visual State Space Blocks are neural components that adapt state-space models to 2D visual data using multi-directional, recurrent scans.
• They integrate adaptive gating, input-conditioned recurrences, and feature fusion to efficiently capture both local and global dependencies.
• Applications span image classification, segmentation, detection, and restoration, offering linear computational complexity over traditional methods.

A Visual State Space (VSS) block is a neural architectural component that adapts state-space models (SSMs)—originally devised for sequence modeling—to the spatial structure of visual data. Unlike traditional convolutional or pure attention-based blocks, a VSS block leverages recurrent state-space updates over spatially arranged tokens, typically in multiple directions, to combine linear-time modeling of both local and global dependencies with low computational overhead. Modern VSS blocks serve as the fundamental operator in a wide family of vision backbones, including Mamba-based architectures, and underpin diverse applications such as classification, segmentation, detection, compression, and restoration across both natural and remote sensing imagery.

1. Mathematical Foundations and Core Variants

VSS blocks are rooted in the linear state-space model, which in continuous time is given by

$$h'(t) = A h(t) + B x(t), \qquad y(t) = C h(t) + D x(t)$$

where $x(t)$ is the input, $h(t)$ is a hidden state, and $y(t)$ is the output. For deep learning, this model is discretized (typically by zero-order hold) to obtain

$$h_k = \bar{A} h_{k-1} + \bar{B} x_k, \qquad y_k = C h_k + D x_k$$

with recurrent parameter matrices $\bar{A}$, $\bar{B}$, $C$, $D$. In a visual setting, the 2D feature map (e.g., $X \in \mathbb{R}^{B\times C\times H\times W}$) is flattend into a long sequence, and the SSM is applied along various spatial orderings.

Recent designs implement these recurrences in a selective or input-conditioned manner (as in Mamba/SS2D), so $A,B,C$ (and sometimes their step sizes) are locally modulated per spatial position using lightweight learned networks. The majority of VSS blocks incorporate multi-way scanning—executing the SSM forward pass along four or more directions (row, column, diagonals), with scanned outputs fused by summation or learned gating (Liu et al., 2024).

Notable variants extend the canonical SSM scan with:

• Gated connections (e.g., residual or context mask gates (Haruna et al., 27 Mar 2025), sigmoid-modulated outputs)
• Deformable or non-raster path scans that allow dynamic, input-conditioned spatial routes (Liu et al., 8 Apr 2025)
• Hybridization with attention, adding a multi-head self-attention stream and fusing with SSM-derived features (Haruna et al., 27 Mar 2025)
• Non-causal (global pooling-like) contraction, producing non-directional, permutation-invariant mixing (Shi et al., 2024)
• Channel-averaging compression, reducing the SSM to 1D along the channel mean for hardware efficiency (Chi et al., 2024)

2. Architectural Integration and Block Structure

A canonical VSS block consists of the following high-level structure:

• Input normalization (optional LayerNorm or batch normalization)
• Primary SSM/SS2D operator: executes multi-way linear SSM recurrences (e.g., four “scan” directions—row-wise, col-wise, reverse row, reverse col), each mapping the flattened spatial tokens through input-conditioned state transitions (Liu et al., 2024).
• Gating/attention module: modulates SSM outputs via input-adaptive gates, context masks, or explicit attention pools.
• Branching and Feature Fusion: Some variants (e.g., vGamba (Haruna et al., 27 Mar 2025)) split channels or features, compute SSM and attention outputs in parallel, then merge via learned gates. Group-based variants (e.g., GroupMamba (Shaker et al., 2024)) process different channel groups along separate scan directions with channel modulation.
• Feed-forward head: 1×1 and/or depthwise convolutions and nonlinearities (GeLU, SiLU) further mix channels.
• Residual connection: the block output is typically the sum of the input and a transformed (often gated) combination of SSM and/or attention features.
• Specializations include conditional parameterization (e.g., C-VSS with image-type–conditioned LayerNorm and gates (Hosseini et al., 2024)), integration of local bias samplings (MultiDW, deformable convs), or frequency-domain mixing modules.

3. Computational Complexity and Efficiency

The principal advantage of VSS blocks lies in their linear computational complexity with respect to the number of spatial tokens, contrasted with the quadratic complexity of multi-head self-attention:

• Four-way SS2D scan: $O(HW\,d^2)$, where $d$ is the channel or hidden state size. Each scan direction flattens and runs an SSM recurrence of linear length.
• Standard 2D convolution: $O(HW\,C_{in}\,C_{out}\,K^2)$ for kernel size $K$.
• Full MHSA (transformer block): $O(N^2 d)$ for $N=H\,W$ spatial locations, which becomes intractable for high-resolution inputs.
• Hybrid SSM+MHSA blocks combine $O(Nd^2)$ (SSM) and $O(N^2 d)$ (attention); gating tends to keep the total cost lower than pure transformer blocks (Haruna et al., 27 Mar 2025).
• Group- and mean-based compression: Dividing SSMs by channel groups or collapsing the channel dimension reduces parameter count and FLOPs by up to 4× (group) or $\sim$D-fold (mean) (Shaker et al., 2024, Chi et al., 2024).

This results in architectures that scale linearly with image size, with demonstrated empirical speedups (VMeanba: up to 1.12× speedup at ≤3% accuracy loss (Chi et al., 2024); GroupMamba: 26–36% parameter reduction with higher or equal accuracy (Shaker et al., 2024)).

4. Specialized Mechanisms: Deformable, Non-Causal, and Hybrid Blocks

Modern VSS blocks adopt several advanced mechanisms:

• Deformable scanning (DefMamba/DM block): Dynamic prediction of spatial offsets and token permutations, enabling spatially structure-aware feature prioritization through end-to-end learned offset networks and attention to semantic regions (Liu et al., 8 Apr 2025).
• Atrous-Window selective scan (AWVSS): Local SSM scans over multi-scale, dilated windows, bridging local detail with global receptive field—especially crucial for dense prediction and change detection (Wang et al., 22 Jul 2025).
• Non-causal duality (NC-SSD): Discards the recurrence magnitude to yield a permutation-invariant, global mixing operator, implemented by contracting the sequence along scalar self-weights and then broadcasting back to tokens; this achieves fully parallel, linear-time aggregation (Shi et al., 2024).
• SS2D with conditional adaptation (C-VSS): Per-dataset/image-type learned scaling, shifting, and gating of VSS activations for dynamic normalization/attention in mixed-domain tasks (Hosseini et al., 2024).

5. Applications, Empirical Results, and Advantages

VSS blocks have been deployed in a wide spectrum of vision tasks:

• Image classification: VMamba, VSSD, GroupMamba, DefMamba, VMeanba, and vGamba achieve 81–84% top-1 accuracy on ImageNet-1K with model sizes and FLOPs competitive with or below Swin/ConvNeXt (Liu et al., 2024, Haruna et al., 27 Mar 2025, Shaker et al., 2024, Shi et al., 2024, Liu et al., 8 Apr 2025, Chi et al., 2024). GroupMamba achieves 83.3% top-1 with 23M parameters and ∼26% efficiency gain over existing Mamba designs (Shaker et al., 2024).
• Segmentation and Detection: High-resolution segmentation on ADE20K and COCO detection tasks shows that VSS-based models outperform or match classical and ViT backbones (e.g., HRVMamba achieves 79.4–80.2% mIoU on Cityscapes, 43.5% on PASCAL-Context (Zhang et al., 2024); vGamba-B achieves 50.9 mIoU at 941G FLOPs (Haruna et al., 27 Mar 2025)).
• Visual restoration/compression: MambaVC leverages 2D-SSM scanning in VSS blocks to outperform both CNN and Transformer-based learned compressions by 9.3% and 15.6% BD-rate, with substantial reductions in compute and memory (Qin et al., 2024). In deblurring, VSS blocks with geometric transforms beat FFTformer and Restormer with much lower cost (Kong et al., 2024).
• Saliency prediction and cross-domain modeling: C-VSS conditional adaptation yields universal state space blocks that adapt normalization and attention parameters for each image modality/task with consistent performance gains (Hosseini et al., 2024).
• Remote sensing and change detection: Dual-branch (CNN + VSS) networks and atrous-window scanning in RS3Mamba, AWMambaBCD/SCD, and HRVMamba enable state-of-the-art accuracy and resolution efficiency for segmentation and land-use change tasks (Ma et al., 2024, Wang et al., 22 Jul 2025, Zhang et al., 2024).

6. Interpretations, Limitations, and Future Directions

VSS blocks inherit the global receptive field capacity of architectural models like ViT but do so with linear complexity. Deep ablation studies repeatedly confirm that:

• Multi-route scanning (≥4 directions) is essential for full spatial information coverage and stable training (Liu et al., 2024, Qin et al., 2024).
• Channel grouping and mean collapse can reduce compute without significant loss in accuracy, provided layer selection is judicious (Chi et al., 2024).
• Deformable and non-causal variants yield improvements in localization, semantic structure, and dense prediction, but introduce additional hyperparameters and potential for adaptation artifacts (Liu et al., 8 Apr 2025, Shi et al., 2024).
• The SSM scan’s effectiveness in very high-resolution or data-regime-specific settings may require hybridization with attention (as in vGamba or C-VSS) for maximal accuracy (Haruna et al., 27 Mar 2025, Hosseini et al., 2024).

Current limitations include the complexity of dynamic offset networks (DefMamba), reliance on accurate parameterization and initialization (e.g., weighting vectors in NC-SSD (Shi et al., 2024)), and susceptibility to adversarial attacks targeting the scan pattern or channel structure (BadScan (Deshmukh et al., 2024)). Ongoing research is investigating more efficient hardware utilization (e.g., VMeanba’s single-channel scan (Chi et al., 2024)), improved fusion with convolutional modules (HRVMamba (Zhang et al., 2024)), and universal adaptability (SUM (Hosseini et al., 2024)).

The VSS block paradigm is now a central operator for efficient, scalable, and context-rich vision architectures, with evidence supporting its superiority over both pure convolutional and attention-based designs across multiple vision domains.