SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models

Abstract: Large Vision-LLMs (LVLMs) have achieved remarkable progress in multimodal understanding, yet their enormous parameter scale and cross-modal computation incur substantial memory and latency overhead, severely limiting real-world deployment on resource-constrained devices. Binarization offers an attractive solution by drastically reducing storage and computational costs. However, existing binarization methods neglect the varying importance of weights across different layers and modalities. This causes parameters irrelevant to downstream tasks to be unnecessarily retained, whereas modality-critical weights may not be adequately optimized, resulting in significant performance degradation. To address these challenges, we develop a novel \underline{S}ignificance-\underline{A}ware \underline{B}inarization for \underline{L}arge \underline{V}ision-\underline{L}anguage \underline{M}odels (SAB-LVLM). Specifically, after constructing Hessian matrices for textual and visual inputs, we propose a spatial significance map to distinguish full-precision weights activated under a single modality from those activated across modalities. We then devise a modality-guided integration strategy to obtain the significance-aware binarization map, which measures weight significance across layers and modalities. Subsequently, this binarization map is incorporated into the binarization objective as an error reweighting term, and binarization fitting is performed through an alternating significance-weighted update scheme. Extensive experiments illustrate the superiority of our SAB-LVLM over existing binary PTQ methods under an approximately 1-bit compression constraint. Our code is accessible at https://github.com/LyuQi127/SAB_LVLM.

• The paper introduces a significance-aware 1-bit binarization framework that enhances LVLM performance while reducing memory and latency costs.
• It employs Hessian-based sensitivity and modality-guided integration to differentiate important unimodal and cross-modal weights for optimized quantization.
• Experimental results show up to 14-point improvements over binary baselines, maintaining critical multimodal capabilities on diverse benchmarks.

Significance-Aware Binarization for Large Vision-LLMs: SAB-LVLM

Introduction

Large vision-LLMs (LVLMs) have established strong multimodal capabilities but their immense parameter scale and associated cross-modal computation result in prohibitive memory and latency costs, severely impeding deployment on resource-constrained devices. While post-training quantization (PTQ) methods offer substantial compression, existing binary PTQ approaches suffer pronounced performance degradation due to their disregard for the heterogeneous and modality-specific significance of weights, particularly within the LVLM architecture. The paper "SAB-LVLM: Significance-Aware Binarization for Large Vision-LLMs" (2607.01876) systematically addresses this deficit by introducing a binarization framework that explicitly models and exploits the modality- and layer-wise variations in parameter importance.

Methodological Framework

The SAB-LVLM framework introduces a set of algorithmic contributions designed to enable effective 1-bit quantization for LVLMs without compromising task-relevant performance. These contributions include:

1. Spatial Significance Map Construction: The approach utilizes Hessian-based sensitivity calculations for visual and textual modalities using calibration data. Analyses of the resulting gradients yield a spatial significance map ($\Gamma$) that quantifies the degree to which individual weights are activated by unimodal or cross-modal inputs, discriminating between weights critical for unimodal versus multimodal operation.
2. Modality-Guided Significance Integration (MGSI): A modality integration score ($r$) is adaptively estimated for each layer, capturing the preference distribution of task relevance across modalities and fusing local and global significance information into a final binarization map ($\Lambda$). The result is an error reweighting scheme that upregulates quantization fidelity on functionally critical subspaces.
3. Alternating Significance-Weighted Update: Building on the error-weighted objective, SAB-LVLM deploys a multi-stage alternating update procedure—akin to ARB-LLM strategies but modulated by per-weight significance estimations—allowing prioritized fitting of binary parameters subject to their contextual importance.

   Figure 1: Architectural pipeline of SAB-LVLM, highlighting spatial significance calculation, modality-guided integration, and the alternating significance-weighted update.

The theoretical justification is anchored in an error allocation result (Theorem 1) that asserts, under a relaxed budget constraint, the optimal quantization error is inversely proportional to the local significance defined in $\Lambda$. This formalizes the intuition that task-relevant weights (those with high $\Lambda_{ij}$) are better preserved during binarization.

Figure 2: (a) SAB-LVLM compared to prior PTQ and binarization baselines; (b) visualization of the spatial significance map for visual and textual activations; (c) performance-memory tradeoff on the MMStar benchmark.

Experimental Analysis

Benchmarks and Model Variants

Comprehensive evaluation was conducted on Qwen2.5-VL and InternVL3.5 families (7B to 72B parameters), across canonical multimodal tasks (MMStar, DocVQA, TextVQA, Video-MME, VSI-Bench). Results are compared with state-of-the-art PTQ baselines including GPTQ (3-bit), PB-LLM, BiLLM, and ARB-LLM (1-bit).

Quantitative Results

SAB-LVLM consistently establishes new Pareto frontiers under $\sim$1-bit constraints, yielding large improvements over existing binarization. On Qwen2.5-VL-7B/32B/72B, it achieves absolute gains of 7–14 points on MMStar over the strongest binary baselines and markedly narrows the gap to 16-bit full-precision. Similar trends are observed on InternVL3.5 variants.

Ablation studies confirm:

• The effectiveness of the joint spatial significance and MGSI, especially the necessity of using modality-adaptive scoring rather than fixed or unimodal-only metrics.
• The optimal binarization performance is highly sensitive to the thresholding in the spatial significance map, supporting the importance of precise partitioning between single-mode and cross-modal weights.

Strong performance is retained across all major capability axes, with coarse and fine-grained perception, logical reasoning, and visual spatial intelligence tasks all exhibiting significant robustness to 1-bit quantization when using SAB-LVLM. Qualitative analyses confirm improved answer consistency over prior approaches in highly multimodal queries.

Figure 3: Qualitative performance analyses on MMStar with Qwen2.5-VL-7B-Instruct, highlighting complex perception, counting, and reasoning questions.

Figure 4: Visualization of the dynamic, layer-wise modality integration score $r$ among self-attention and MLP down-projection layers, confirming distributional variability.

Theoretical and Practical Implications

SAB-LVLM sets an explicit curriculum for future LVLM binarization: quantization performance is decisively improved by embedding high-granularity, modality- and layer-aware weight significance modeling directly into the error objective and the fitting process. The explicit modeling of cross-modal weight contributions addresses a central limitation in previous generic PTQ methods, enabling the robust application of binarization at scale to multimodal architectures.

Practically, this advances the deployability of high-performing LVLMs on constrained hardware by an order of magnitude relative to naive binarization, substantially reducing parameter storage and inference latency while largely retaining critical perceptual and reasoning capacities.

Prospects and Future Directions

Given the generality of the significance-aware binarization paradigm, several future research directions are immediately evident:

• Extension to broader LVLM architectures (including those with complex cross-modal adapters, attention routing, or multi-stage integration).
• Application to settings requiring rigorous temporal/contextual reasoning, such as long-context video understanding or embodied multimodal interaction.
• Exploration of significance-driven quantization beyond binarization, such as adaptive-mixed precision, which could further optimize the efficiency-accuracy frontier.

Conclusion

The SAB-LVLM framework introduces a principled, highly effective approach for 1-bit quantization in large vision-LLMs, driven by explicit modeling of modality- and layer-specific weight significance. Its integration of Hessian-based sensitivity analysis, adaptive modulation by cross-modal importance, and an alternating optimization scheme sets a new standard in binary PTQ for LVLMs. The resulting compression gains and retention of downstream performance open new avenues for energy- and memory-efficient deployment of advanced multimodal AI.