Arrangement-Specific Lightweight MLPs

• Arrangement-specific lightweight MLPs are modular, parameter-efficient architectures that exploit spatial and arrangement priors through slicing, grouping, and rearrangement techniques to reduce computational cost.
• They integrate methods such as spatial axis factoring, deformable mixing, and segmented channel mixing across vision, language, and audio domains to balance local and global feature processing.
• Empirical evaluations demonstrate competitive accuracy with reduced parameter counts and FLOPs compared to dense MLPs and transformer layers, underscoring their practical efficiency.

Arrangement-specific lightweight MLPs are a class of modular, parameter-efficient multilayer perceptron architectures that exploit spatial, structural, or task-specific arrangements within input or feature tensors to reduce computational cost, parameter count, or both, while maintaining or minimally degrading model performance. These architectures deploy explicit or implicit modularization—via slicing, grouping, spatial rearrangement, axial splits, or learnable geometric bias—to isolate localized computations, restrict global mixing, or encode arrangement priors in a lightweight manner. Techniques are implemented across natural language, vision, and audio domains, frequently serving as computationally efficient alternatives to dense MLP or attention layers in deep neural networks.

1. Core Mathematical and Architectural Principles

Arrangement-specific lightweight MLPs implement a range of modularization strategies that exploit known or designed tensor arrangements:

• Static Modularization via Slicing and Summation:

MLPMoE deterministically partitions the intermediate dimension of a standard transformer MLP into $B$ contiguous blocks (experts). Algebraically, for input $x \in \mathbb{R}^{d_\text{model}}$, the transformation is:

$$y = \sum_{b=0}^{B-1} a_b W_\text{down}^{(b)} \big( \phi(W_\text{gate}^{(b)} x) \odot W_\text{up}^{(b)} x \big)$$

with branch-specific slices. The regrouping is functionally identical to a standard MLP but exposes explicit sparsity and topological modularity (Novikov, 26 Nov 2025).

• Spatial Axis Factoring:

Architectures such as RaftMLP replace a global token-mixing MLP (mixing across all spatial locations) with two distinct and smaller MLPs, one acting along the height (vertical mixing) and one along the width (horizontal mixing). The process leverages arrangement-induced inductive bias, enabling lower parameter overhead and computational complexity:

$$Z^\text{ver} = X + W_{2,\mathrm{ver}}\,\mathrm{GELU}(W_{1,\mathrm{ver}}\,X)$$

$$Z^\text{hor} = Z^\text{ver} + W_{2,\mathrm{hor}}\,\mathrm{GELU}(W_{1,\mathrm{hor}}\,Z^\text{ver})$$

with additional grouping of channels into local "rafts" for denser localized processing (Tatsunami et al., 2021).

• Local Grouping and Deformable Mixing:

SpiralMLP introduces Spiral FC layers, instantiated as fixed-offset deformable convolutions, sampling features along a spiral path through a 2D feature map rather than axes or full global flattenings:

$$\mathrm{SpiralFC}_{i,j,:}(X)=\sum_{c=0}^{C_{\rm in}-1} X_{i+\phi_i(c),j+\phi_j(c),c} W^{\rm spiral}_{c,:} + b^{\rm spiral}$$

This framework enables balancing local and midrange context via controlled spiral radius and partitioning (Mu et al., 2024).

• Hierarchical Spatial Rearrangement:

Hire-MLP uses hierarchical region partitioning and rearrangement within a feature map. By flattening and mixing within localized spatial partitions, followed by cross-region circular shifts, the MLP captures both local and global spatial context without abandoning the underlying spatial structure (Guo et al., 2021).

• Segmented Channel Mixing:

SplitMixer reduces the cost of per-layer channel mixing by splitting channels into smaller overlapping or non-overlapping segments and mixing only within those; spatial mixing is addressed using sequences of 1D depthwise convolutions along separate axes, further reducing parameter and FLOP utilization (Borji et al., 2022).

• Adapter MLPs for Layout Priors:

In context-aware frameworks such as VICL, tiny two-layer MLP adapters (e.g., 0.8M parameters for a 768×512 bottleneck) are appended for each distinct arrangement of input regions (e.g., grid layouts in in-context learning). Each adapter is responsible for encoding geometry-specific priors, with the main backbone (e.g., a frozen MAE-VQGAN) focusing on high-level semantics (Liao et al., 15 Jan 2026).

2. Structured Sparsity and Pruning Mechanisms

Several arrangement-specific MLP frameworks implement explicit or implicit sparsity via parameter pruning or routing control:

• Fractal Fade (MLPMoE):

Implements a differential sparsity schedule across branches: for branch $i$, target sparsity is $s_i = q \cdot \frac{i}{B-1}$. For each parameter matrix $M$ in a branch, all elements below the empirical $s_i$-quantile of $|M|$ are zeroed, encoding a fractal sparsity pattern that leaves earlier branches dense and prunes later ones progressively more aggressively. The result is an ≈18–20% drop in nonzero parameter count with minimal perplexity degradation in LLMs (Novikov, 26 Nov 2025).

• Compensated Pruning:

Branches beyond a cutoff $K < B$ are dropped by setting their gating $a_b=0$ and scaling surviving branches by $a_b = \frac{B}{K}$, which preserves overall output variance. This statically removes a controlled fraction of computation while maintaining functional alignment with the unpruned network (Novikov, 26 Nov 2025).

• Channel/Group-level Sparsification (SplitMixer, SV-Mixer):

By restricting mixing to smaller segments (SplitMixer) or channel groups (SV-Mixer), entire blocks of the parameter matrix are structurally zero outside of local blocks, further reducing parameter and compute requirements (Borji et al., 2022, Heo et al., 17 Sep 2025).

3. Domain-specific Instances and Efficiency Outcomes

Arrangement-specific lightweight MLPs are deployed in language, vision, and audio domains:

• Language (MLPMoE):

Applied post hoc to LLMs (e.g., Qwen2.5-0.5B-Instruct, DeepSeek-R1-Distill-Llama-8B), MLPMoE achieves ≤0.05% proxy perplexity degradation in the unsparsified case and ≈2% degradation with ≈20% of MLP parameters pruned. The transformation does not require retraining, calibration, or router optimization, and parameter count remains effectively constant unless pruning is activated (Novikov, 26 Nov 2025).

• Vision (RaftMLP, SpiralMLP, Hire-MLP, SplitMixer):

RaftMLP achieves 76.1%/78.8%/79.4% top-1 accuracy on ImageNet-1K for S/M/L variants with parameter budgets substantially below corresponding MLP-Mixer, ResMLP, or gMLP baselines. SpiralMLP (e.g., SpiralMLP-B5) achieves 84.0% on ImageNet-1k with 68M parameters and 11.0G FLOPs, matching heavyweight transformer and CNN models with lower compute (Mu et al., 2024). Hire-MLP attains 83.8% ImageNet-1K top-1 in the Large setting, 51.7 box AP and 44.8 mask AP on COCO val2017, and 49.9 mIoU on ADE20K (Guo et al., 2021). SplitMixer-I (256/8) achieves 93.91% on CIFAR-10 at 0.28M parameters—a ≈2× reduction from ConvMixer with equivalent accuracy (Borji et al., 2022).

• Audio (SV-Mixer):

SV-Mixer replaces transformer encoders in SSL speaker verification pipelines, using Local-Global, Multi-Scale, and Group Channel Mixing modules. An MLP-based SV-Mixer encoder block uses ≈55% fewer parameters and ≈50% fewer GMACs versus a transformer encoder, outperforming a transformer student by 14.6% relative (EER=1.52% vs 1.78%) while closing within 0.01% of the teacher at 75% compression (Heo et al., 17 Sep 2025).

• Arrangement-specific Adapters (VICL):

In visual in-context learning, up to eight 2-layer bottleneck MLP adapters (≈0.8M parameters each) encode spatial priors tied to the arrangement of tiled query/support regions. Integrating these adapters yields 3.7-point mIoU gain in segmentation over prior methods, with only 0.0021 GFLOPs additional compute (vs. 13.27 GFLOPs backbone) and near-zero runtime or memory impact (Liao et al., 15 Jan 2026).

4. Design Trade-offs and Implementation Considerations

Critical factors in deploying arrangement-specific lightweight MLPs include:

• Branch/Segment/Region Granularity:

Choice of the number and size of branches (MLPMoE), channel segments (SplitMixer), or spatial partitions (Hire-MLP, RaftMLP) impacts both model expressivity and efficiency. Finer splits generally yield sparser, more modular computation at the expense of residual or aggregation overhead.

• Sparsity Schedules:

Fractal Fade’s linear schedule $s_i = q \cdot \frac{i}{B-1}$ can be replaced with exponential or fractal schedules to adapt pruning to layer or region sensitivity; in vision models, excessively large partitions can harm local detail, while regions too small miss global context (Novikov, 26 Nov 2025, Guo et al., 2021).

• Static vs. Dynamic Routing:

Static modularization and sparsity (as in MLPMoE or vision MLPs) offer training- and calibration-free deployment. However, lightweight router modules can be attached for conditional activation, at the cost of additional calibration (Novikov, 26 Nov 2025).

• Hardware/Kernel Support:

Many arrangement-specific MLPs require custom block-sparse or masked computation kernels for optimal deployment efficiency, as standard deep learning frameworks may not skip zeroed weights or non-invoked branches, increasing wall-clock latency (Novikov, 26 Nov 2025).

• Fine-tuning:

Zero-shot conversion is typically effective, yet slight degradation can be addressed via branch- or region-level fine-tuning, recovering accuracy after modularization or pruning (Novikov, 26 Nov 2025).

5. Empirical Evidence and Comparative Performance

Across application domains, arrangement-specific lightweight MLPs demonstrate competitive or superior efficiency-accuracy trade-offs relative to dense MLPs, convolutional, or transformer baselines:

Model	Params (M)	FLOPs (G)	Top-1 (%)	Reference
RaftMLP-S	9.9	2.1	76.1	(Tatsunami et al., 2021)
SpiralMLP-B5	68	11.0	84.0	(Mu et al., 2024)
Hire-MLP-Large	96	13.4	83.8	(Guo et al., 2021)
SplitMixer-I (256/8)	0.28	71M	93.9*	(Borji et al., 2022)
MLPMoE-8B, unsparse	8,030	—	<0.05% PPL increase	(Novikov, 26 Nov 2025)
SV-Mixer student	—	—	1.52 EER (Vox1-O)	(Heo et al., 17 Sep 2025)

*SplitMixer reports CIFAR-10 accuracy.

These results indicate that properly designed arrangement-specific modularization, especially when leveraging native geometric, channel, or temporal structure, preserves or improves performance at substantially reduced parameter and computational budgets.

6. Encoded Priors, Adaptation, and Domains of Applicability

Arrangement-specific lightweight MLPs explicitly encode geometric, spatial, temporal, or arrangement priors:

• Explicit Geometry (Vision):

Axis splits (RaftMLP), spiral trajectories (SpiralMLP), region hierarchies (Hire-MLP), and arrangement-specific adapters (VICL) capture inherent image structure, support for variable resolutions, and flexible adaptation to new layouts (Tatsunami et al., 2021, Mu et al., 2024, Guo et al., 2021, Liao et al., 15 Jan 2026).

• Arrangement Priors (In-Context Learning):

In the VICL framework, adapters encode the latent impact of support/query positioning, decoupling layout-specific reasoning from the frozen content backbone (Liao et al., 15 Jan 2026).

• Spectrotemporal Decomposition (Audio):

Multi-scale, group-channel, and local-global mixing in SV-Mixer exploits the structured arrangement of acoustic features over time and frequency, enabling deep compression and efficient distillation (Heo et al., 17 Sep 2025).

• Transformers/LLMs:

The MLPMoE approach demonstrates that explicit algebraic decomposition along tensor axes exposes semantically meaningful “experts” and admits efficient static pruning and modularization that would be inaccessible in monolithic dense feed-forward designs (Novikov, 26 Nov 2025).

7. Limitations, Open Directions, and Optimal Deployment

• Arrangement Designs Are Use-case Specific:

The effectiveness of modularization is domain- and task-dependent. For instance, full global token mixing remains irreplaceable in some domains, and naive channel splitting may harm cross-segment information flow if not sufficiently deep or overlapping (Borji et al., 2022).

• Inference Kernel Optimization Required:

To leverage real-world speedups, deployment stacks must support block-sparse or branch-skipping execution; linear algebra libraries without such support yield only theoretical efficiency (Novikov, 26 Nov 2025).

• Search and Fine-tuning Overhead:

Some approaches (VICL's arrangement MLPs) require arrangement validation and adapter selection, yielding small, per-experiment fine-tuning overhead, though parameter count remains marginal relative to backbones (Liao et al., 15 Jan 2026).

• Potential for Dynamic and Conditional Extensions:

Conditional computation (e.g., via routers) is compatible and may further enhance efficiency. Static designs provide immediate hardware friendliness and explainability, but may be suboptimal for certain data distributions (Novikov, 26 Nov 2025).

• Layer-wise Sensitivity Variation:

Aggressive pruning or excessive modularization may compromise sensitive layers, suggesting that per-layer tuning of module granularity and pruning parameters is preferable to global heuristics (Novikov, 26 Nov 2025, Guo et al., 2021).

A plausible implication is that as model and deployment complexity increase, arrangement-specific lightweight MLPs provide a scalable path to modularity, compression, and domain specialization with minimal accuracy loss and hardware-friendly execution profiles.