LASER: A Corrective Lens for LVLMs via Visual Attention Preservation and Sink Suppression

Abstract: Large vision-LLMs (LVLMs) exhibit strong reasoning ability but suffer from visual forgetting during long-horizon decoding, where attention progressively drifts away from visual evidence. Existing methods largely treat this issue as a late-stage attention decay problem or attempt to mitigate it through heuristic reminders or post-hoc attention lifting. Through systematic empirical analysis, we find that performance degradation under visual forgetting is largely driven by two overlooked factors: early-stage attention decay disrupts evidence acquisition, and attention concentration on a subset of task-irrelevant visual sink tokens. Motivated by these insights, we propose LASER, a post-training framework that regulates both the visual attention trajectory and intra-visual token attention distribution during reasoning. Technically, LASER introduces two complementary rewards: a Visual Grounding Reward, which encourages the model to maintain attention on semantically salient visual tokens throughout decoding, and a Sink Suppression Reward, which penalizes excessive attention concentration on visual sink tokens. Together, these rewards preserve early-stage grounding while preventing attention collapse onto uninformative regions. Extensive experiments on eight benchmark datasets demonstrate that LASER consistently outperforms strong baselines, validating attention-aware training as an effective remedy for visual forgetting.

• The paper introduces LASER, a reward-shaping framework that preserves visual attention and suppresses distracting sink tokens in LVLMs.
• It employs Group Relative Policy Optimization to maintain early-stage visual grounding and reallocate attention, achieving up to a 12.7% performance gain.
• Empirical results validate LASER's robust improvements across multimodal benchmarks without inflating response length or computational overhead.

LASER: Attention Correction for LVLMs via Visual Grounding and Sink Suppression

Introduction and Motivation

Large Vision-LLMs (LVLMs) equipped with chain-of-thought (CoT) reasoning capabilities demonstrate substantial potential for complex multimodal task solving. However, extended-generation scenarios reveal a deficiency: visual forgetting. Attention to visual inputs dissipates as output sequences grow, causing the model to inappropriately rely on self-generated textual context and reducing reasoning quality for visually-grounded inference.

Figure 1: Visual forgetting during long-horizon reasoning in LVLMs—the attention to visual tokens decays (middle), leading to poor focus on task-relevant visual regions during later decoding (right).

Current attempts to mitigate visual forgetting—such as late-stage attention boosting and visual reminder injection—primarily target the tail-end of reasoning, overlooking the importance of early-stage grounding and the distributional collapse of attention onto visual sink tokens: semantically irrelevant but attractively attended image regions. These factors undermine both evidence accumulation and semantic grounding across the reasoning chain.

Analysis of Visual Forgetting in LVLMs

The paper presents a systematic diagnosis of visual forgetting, highlighting two key findings:

1. Early-stage attention loss is especially detrimental. Interventions that suppress visual attention early in generation induce far greater reasoning degradation than identical interventions at later steps. This demonstrates the outsized importance of early visual grounding in constructing reliable reasoning trajectories.
2. Attention distribution collapses onto visual sink tokens. Persistent subsets of visual tokens absorb the majority of attention across reasoning steps, regardless of their semantic irrelevance. Quantitative interventions reveal that redistributing attention away from these sink tokens—rather than uniformly scaling visual token influence—yields consistent gains in reasoning accuracy.

   Figure 2: Visualization showing attention focused on semantically uninformative image regions (“sink tokens”) over reasoning steps.

   

Figure 3: Left—Redistributing attention from sink tokens consistently improves accuracy. Right—Sink attention ratio remains high for standard RL training.

These findings motivate a shift from magnitude-only or stage-specific interventions to trajectory-level regulation of attention, both in temporal (when) and distributional (where) terms.

The LASER Framework

Method Overview

LASER (LVLM Attention-aware Sight-Enhanced Reasoning) introduces a reward-shaping framework applied during post-training via Group Relative Policy Optimization (GRPO). Unlike inference-time modifications, LASER enforces persistent adjustment of attention dynamics. Its methodology consists of:

• Visual Grounding Reward ($R_\text{vis}$): Encourages attention maintenance on non-sink, semantically informative tokens across all generation steps. This component suppresses premature disengagement from visual evidence and ensures early-stage semantic grounding.
• Sink Suppression Reward ($R_\text{supp}$): Penalizes excessive attention to sink tokens, thus dispersing focus toward task-relevant image regions and alleviating distributional collapse.

  Figure 4: LASER computes complementary rewards from attention maps during generation: $R_\text{vis}$ (grounding), $R_\text{supp}$ (sink suppression).

Visual sink tokens are identified via activation analysis of specific anomalous dimensions in the model’s hidden state, following cited best practices targeting persistent, model-agnostic sink behaviors.

The combined reward, gated by output correctness, forms the total loss for GRPO, facilitating trajectory-level attention optimization throughout reasoning.

Empirical Results

LASER is rigorously benchmarked on eight datasets spanning mathematical reasoning (MathVista, MathVerse, MathVision, WeMath), general multimodal reasoning (MMMU, MMStar, LogicVista), and visual perception (HallusionBench). Qwen2.5-VL-7B-Instruct serves as the primary base, with comparisons to leading proprietary (GPT5, Gemini 2.5 Pro) and open-source models, including specialized attention-control baselines.

Key numerical results include:

• Consistent performance gains: LASER demonstrates up to 12.7% absolute improvement in MathVision, 3.3% on HallusionBench, and 1.4–9.3% on other challenging benchmarks versus VL and RL-based competitors. The margin is largest for tasks demanding sustained visual grounding.
• LASER outperforms reflection- and VAPO-based attention repair methods across reasoning and perception suites.

LASER’s improvements are not attributable to response length inflation or trivial attention mass increases; rather, qualitative analysis demonstrates sustained, fine-grained visual focus and coherent semantic progression from images to final answers.

Figure 5: Left—LASER suppresses sink token attention throughout generation. Right—Response length remains controlled across training epochs, demonstrating efficient, non-redundant reasoning.

Figure 6: LASER maintains visual attention, yielding correct inference, while the base model disengages and fails on a visual perception task.

Ablation and Analytical Insights

Ablation studies indicate that both $R_\text{vis}$ and $R_\text{supp}$ provide additive improvements on all metrics. $R_\text{vis}$ reduces overall visual forgetting, while $R_\text{supp}$ is crucial for rebalancing attention and preventing spurious focus on uninformative regions. Similar gains are observed when LASER is applied to SFT-based models (Revisual-R1), underscoring robustness to initialization and supervision regimes.

Attention trajectory analysis demonstrates that standard RL-based reasoning yields sharp decay in visual grounding, while LASER stabilizes and elevates VAP across all datasets. The model neither resorts to length inflation nor increased computation at inference, but rather achieves superior performance through attention manipulation toward semantically rich visual tokens.

Implications and Future Perspectives

LASER’s architecture-agnostic, reward-based regulation of internal attention represents a practical and theoretically grounded solution for robust multimodal reasoning. By shifting the focus from late-stage or magnitude-only interventions to trajectory-level attention allocation, LASER exposes fundamental vulnerabilities in current LVLM reasoning protocols.

Implications for both theory and practice include:

• Generalization to larger models and diverse architectures: Given model-agnostic sink-detection and attention-based reward shaping, scaling LASER to higher-parameter LVLMs remains a compelling direction.
• Broader application for hallucination control: The attention-preservation and sink-suppression mechanisms directly target spurious content generation, suggesting relevance for domains such as medical imaging, scientific QA, and vision-intensive agentic workflows.
• Potential for joint optimization across modalities: Future research may enhance LASER by coupling attention rewards with modality-confidence or multi-step introspection signals.

However, adapting sink token identification to novel architectures may require further study, as current methods are partially tied to observed activation patterns specific to existing model families.

Conclusion

LASER introduces a principled, reward-driven approach to visual attention preservation and sink suppression in LVLMs. By explicitly regulating both the temporal and spatial dimensions of attention during long-horizon reasoning, the framework achieves robust improvements over existing baselines on challenging visual reasoning and perception tasks. LASER’s methodology provides new insight into the causes of visual forgetting and establishes a foundation for more reliable, visually grounded AI systems.

Figure 7: Diverse, high-fidelity grounding in qualitative LASER results, consistently relating visual evidence to each reasoning step.