LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter

Abstract: Comparing transformer backbones for image segmentation is confounded: each is paired with a different decoder, recipe, and pretraining, so reported differences rarely reflect the backbone itself. We introduce the Lightweight Universal Mask Adapter (LUMA), a lightweight, backbone-agnostic mask-transformer head that treats any backbone as a black-box feature extractor, letting a set of queries read from its features through cheap cross-attention. LUMA matches the accuracy of EoMT, the state-of-the-art efficient ViT-segmenter, at lower cost, while attaching unchanged to isotropic, hierarchical, convolutional, and mixture-of-experts backbones alike. Holding this head fixed, we benchmark 20 backbones, 11 pretraining schemes and a range of resolutions on ADE20K and Cityscapes under one modern recipe. We find that ``efficient'' token mixers fail to deliver efficiency even at the high resolutions that motivate them, with plain ViT holding the throughput Pareto-front at every resolution. Additionally, the pretraining objective, not the architecture, the lever the field has tuned hardest, governs segmentation quality.

• The paper demonstrates that LUMA, a fixed mask adapter, isolates backbone effects to enable fair, modern segmentation benchmarking.
• LUMA employs a lightweight cross-attention mechanism with a single well-placed query interaction, ensuring compatibility across diverse backbone designs.
• The study reveals that pretraining objectives and compute allocation, rather than novel token-mixer designs, chiefly drive segmentation performance.

LUMA: A Lightweight Universal Mask Adapter for Fair Segmentation Benchmarking

Introduction

The paper "LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter" (2607.00687) systematically addresses the confounded state of transformer backbone evaluations in image segmentation. Prior work fails to isolate the backbone's true influence, as decoder heads, training recipes, and pretraining methods typically vary across benchmarks. To rectify this, the authors introduce LUMA—a minimal, black-box mask-transformer head that attaches unchanged to any backbone, ensuring that performance differences genuinely reflect architecture rather than confounding factors.

LUMA is specifically designed to decouple the mask transformer head from backbone internals without sacrificing accuracy or efficiency, generalizing beyond plain ViT to isotropic, hierarchical, convolutional, and mixture-of-experts variants. This design enables, for the first time, a fair, modern protocol to disentangle architectural, pretraining, and resolution contributions in dense prediction.

LUMA Architecture

LUMA embodies a minimalist approach: queries are maintained as an external side-stream and interface with backbone representations solely via lightweight cross-attention layers, followed by reuse of the backbone's MLP blocks for each query update. Crucially, this avoids modifications or entanglement with the backbone's attention mechanism, facilitating genuine agnosticism with respect to token-mixer details.

Figure 1: The LUMA architecture; queries interact with features from each tapped backbone block via cross-attention, using unchanged backbone MLPs.

This approach solves the key limitations of EoMT, which inherently presumes globally uniform attention, intervention in block internals, and fixed embedding dimensions. LUMA, in contrast, dynamically aligns query representation dimensions by routing them through the same width-change operations as the backbone, ensuring compatibility regardless of design.

An explicit ablation shows that increasing the number of cross-attention taps degrades task performance, signifying that a single well-placed interaction is optimal—validating the minimalism in LUMA's design.

Figure 2: Ablation demonstrates that increasing cross-attention taps reduces ADE20K mIoU, and 100 queries (EoMT default) yields optimal performance.

Benchmarking Backbones: Architecture and Efficiency

With LUMA as the fixed head, the paper benchmarks 20 distinct transformer backbone architectures, representing the taxonomy of efficient models: vanilla ViTs, low-rank, sparse, kernel-based, fixed, and non-attention mixers, as well as hybrid and convolutional variants.

Segmentation performance measured under LUMA shows that—contrary to common claims—novel "efficient" token-mixers do not deliver significant throughput or accuracy gains in the high-resolution regimes where segmentation operates.

Figure 3: ImageNet top-1 accuracy provides only a moderate correlation with segmentation mIoU on ADE20K, with many convolutional hybrids as significant outliers.

Figure 4: Throughput versus ADE20K mIoU grouped by backbone family; plain ViT (with FlashAttention) consistently defines the Pareto-optimal throughput-accuracy boundary.

Figure 5: Inference memory versus Cityscapes mIoU reveals plain ViT as Pareto-optimal for large models, whereas sparse and non-attention models excel at lower memory budgets.

Backbones such as Linformer, Synthesizer, and MLP-Mixer, which demonstrate robust transfer at classification resolutions, fail catastrophically at segmentation-scale input (e.g., Linformer drops from 38.4 to 3.8 mIoU as resolution increases). This collapse is not exposed by typical classification benchmarks, highlighting the importance of fair, task-specific evaluation.

Overall, plain ViT—with or without FlashAttention—remains throughput- and memory Pareto-optimal for large models, with sparse attention (e.g., Swin, WaveViT) and certain non-attention designs (e.g., FocalNet) as secondary contenders in the efficiency spectrum. Attempts to gain further improvements via changes to the token-mixer are met with diminishing returns.

Pretraining: The Decisive Factor

The extensive study of 11 pretraining objectives—including supervised, contrastive, masked image modeling, self-distillation, and segmentation-specific pretraining—demonstrates that the pretraining objective, not the backbone architecture or ImageNet top-1, fundamentally dictates downstream segmentation quality.

Figure 6: Segmentation mIoU versus ImageNet linear-probe accuracy across pretraining objectives; dense and self-distillation objectives (EVA-02, DINO) outperform supervised even when the latter achieves higher classification accuracy.

Dense objectives such as MAE, EVA-02, and DINO significantly outperform supervised DeiT III and CLIP in segmentation, despite the latter often having higher top-1 or linear-probe scores. Importantly, linear-probe performance decouples from segmentation mIoU as both dense and multimodal objectives (CLIP, SAM) reveal their misalignment with spatial transfer. Supervised pretraining is systematically suboptimal, with up to 8+ percentage points deficit compared to optimal pretraining for the same architecture.

Compute-Optimal Scaling: Balancing Model Size and Resolution

LUMA enables controlled analysis of compute allocation between backbone scaling and input resolution. The empirical accuracy surfaces are well-fitted by a saturating power-law in both parameter count and input size.

Figure 7: Compute-optimal scaling curves (throughput vs. mIoU); the frontier assigns ≈80% of extra compute to increasing model size, with limited gain from excessive resolution scaling.

On both ADE20K and Cityscapes, optimizing for accuracy under a fixed compute budget directs ≈80% of additional computational resources to scaling backbone size, supporting only minor concomitant increases in input resolution until the native image size becomes a constraint. ViT-g is optimal on ADE20K, yet not on Cityscapes—there, increased resolution for smaller backbones (e.g., ViT-B/L at 1024 px) outperforms brute-force scaling of the largest model.

Implications and Future Directions

The rigorous, LUMA-enabled disentanglement of backbone, pretraining, and resolution clarifies longstanding ambiguities in the dense prediction literature. Architectural innovations in token mixing—despite decades of research—offer marginal improvements for segmentation relative to careful selection of pretraining and proper compute allocation.

Key theoretical implications include:

• Token-mixer innovation plateau: The diminishing returns of new mixers in the segmentation regime suggest future architectural advances must target fundamental representational capacity or inductive biases, rather than efficiency alone.
• Pretraining objective as principal driver: Dense-objective pretraining, especially self-distillation and masked modeling, determines the practical upper bound for transfer performance, advocating for future research investment in designing scalable, dense-semantic pretraining objectives.
• Re-evaluating proxy benchmarks: Top-1 and linear-probe metrics are not robust proxies for dense prediction performance, especially as architectures and pretraining diverge; task-aligned benchmarks must become the standard for model selection in dense vision.

Practically, LUMA provides an open gateway for further fair benchmarking, decoupling head design confounds, and directly enabling such studies across future architectural advances and emerging foundation models.

Conclusion

LUMA provides a universal, lightweight mechanism to benchmark segmentation backbones in isolation. The extensive empirical analysis demonstrates that transformer backbone choice, though a focal point of ongoing literature, exerts limited influence on segmentation performance relative to pretraining and scale. Plain ViT (with appropriate engineering such as FlashAttention) consistently defines the efficiency-accuracy Pareto front at large scales. The dominant levers for dense prediction are choice of pretraining objective and resource allocation to backbone size.

Researchers should prioritize robust, segmentation-specific pretraining and reserve efforts in architectural innovation for cases where true novelty in representational universality or efficiency is possible. LUMA offers a fair, minimal-proxy benchmark for future dense prediction research, fostering more meaningful progress in the field.