Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking

Abstract: Large Vision-LLMs (LVLMs) rely on attention-based retrieval of safety instructions to maintain alignment during generation. Existing attacks typically optimize image perturbations to maximize harmful output likelihood, but suffer from slow convergence due to gradient conflict between adversarial objectives and the model's safety-retrieval mechanism. We propose Attention-Guided Visual Jailbreaking, which circumvents rather than overpowers safety alignment by directly manipulating attention patterns. Our method introduces two simple auxiliary objectives: (1) suppressing attention to alignment-relevant prefix tokens and (2) anchoring generation on adversarial image features. This simple yet effective push-pull formulation reduces gradient conflict by 45% and achieves 94.4% attack success rate on Qwen-VL (vs. 68.8% baseline) with 40% fewer iterations. At tighter perturbation budgets ($ε=8/255$), we maintain 59.0% ASR compared to 45.7% for standard methods. Mechanistic analysis reveals a failure mode we term safety blindness: successful attacks suppress system-prompt attention by 80%, causing models to generate harmful content not by overriding safety rules, but by failing to retrieve them.

• The paper introduces an attention hijacking method that bypasses safety instructions in LVLMs, achieving up to 94.4% attack success rate.
• It employs suppression and anchoring losses on the final transformer layers to reduce gradient conflict, leading to 40% faster convergence.
• Empirical results reveal a significant safety blindness in LVLMs, highlighting vulnerabilities in current alignment protocols.

Blinding Large Vision-LLMs to Safety Instructions via Adversarial Attention Manipulation

Introduction and Motivation

Large Vision-LLMs (LVLMs) underpin safety-critical applications through continuous retrieval of alignment-relevant context at each generation step, primarily mediated via attention to prefix (system prompt) tokens. Existing output-based adversarial attacks have demonstrated limited efficiency due to persistent "gradient conflict"—an opposition between adversarial objectives and internal safety-enforcing attention mechanisms. "Seeing No Evil: Blinding Large Vision-LLMs to Safety Instructions via Adversarial Attention Hijacking" (2604.10299) addresses these inefficiencies by directly targeting this internal attention mechanism, proposing an approach that manipulates attention patterns to induce a state termed safety blindness, whereby LVLMs fail to retrieve safety instructions entirely and generate harmful content without resistance.

Problem Diagnosis: Gradient Conflict in LVLM Jailbreaking

Standard adversarial attacks on LVLMs maximize the likelihood of harmful output but are impeded by the model's internal retrieval of safety instructions through persistent attention to prefix tokens. This fundamental misalignment in attack targeting causes high-frequency oscillations and significantly slowed optimization, as measured by negative gradient cosine similarity between adversarial and safety alignment objectives.

Figure 1: Gradient conflict analysis: Baseline attacks exhibit oscillatory convergence dynamics and high rates of severe gradient conflict, while attention-hijacking reduces these conflicts and enables smoother, faster convergence.

Quantitative analysis in the paper reveals that 20% of baseline optimization iterations experience severe gradient conflict (cosine similarity $< -0.5$), which the proposed intervention reduces by 45%.

The Attention-Guided Visual Jailbreaking Method

The method extends standard adversarial optimization with two simple, interpretable auxiliary losses:

1. Suppression Loss: Minimizes attention from generated tokens to alignment-relevant prefix tokens, disrupting safety retrieval at each decoding step.
2. Anchoring Loss: Augments attention toward adversarial image tokens, ensuring the model's generative process is grounded in the compromised visual features.

These are applied over the final $K$ transformer layers, consistent with mechanistic interpretability literature on the localization of refusal behavior.

Figure 2: Method overview. The push-pull mechanism selectively suppresses attention to system instructions and anchors generation on image tokens, leveraging binary position selectors over attention matrices.

This "push-pull" dynamic circumvents, rather than overpowers, the model's alignment protocols, producing efficient and robust adversarial examples.

Empirical Results

The paper demonstrates that attention-guided attacks consistently outperform output-only baselines in attack success rate (ASR) and optimization efficiency:

• On Qwen-VL, the proposed method achieves 94.4% ASR (Llama Guard judge) vs. 68.8% for VAE-JB under comparable settings.
• For LLaVA-1.5-7B, attention-guided optimization yields up to 84.0% ASR, reliably surpassing competing approaches.
• Under stricter $\ell_\infty$ budgets ($\epsilon = 8 / 255$), attention-based attacks remain competitive (59.0% ASR), attesting to their perturbation efficiency and transferability.
• Convergence is achieved in 40% fewer optimization iterations than baselines.

  Figure 3: Attention-guided jailbreaking achieves near-perfect attack success by aligning optimization gradients, effectively blinding LVLMs to safety instructions.

Robustness and Transfer

The proposed attack generalizes across architectures and transfers to closed-source LVLMs (e.g., GPT-4o, Claude-3.5, Gemini-1.5) with higher success than prior attacks, without requiring model-specific reoptimization.

Mechanistic and Causal Analysis

Mechanism: The attack operates by causing a collapse in attention to prefix (system) tokens—termed safety blindness—with an observed 80% suppression of system-prompt attention and a 4.1× amplification of attention to image tokens.

Figure 4: Attention redistribution—system-prompt attention is suppressed (blue) and image token attention is amplified (red) to achieve jailbreak.

Causality: Experimental intervention restoring system-prompt attention in model inference (via positive attention biases) neutralizes the attack without altering the adversarial image, confirming the causal role of attention suppression in enabling harmful outputs.

Ablations and Optimization Characteristics

Layer-wise ablations demonstrate that intervening on the last 6–8 layers is both necessary and sufficient for attack efficacy. The suppression and anchoring losses are complementary, with full performance only achieved by their combination. Perturbation budgets are exploited fully without visible perceptual artifacts—default attacks yield adversarial images with SSIM as high as 0.990.

Figure 5: Ablation analysis shows that attention-guided optimization achieves lower final loss and faster convergence than PGD.

Figure 6: Adversarial images remain visually indistinguishable from originals even at significant perturbation budgets.

Qualitative Output and Failure Modes

Qualitative evaluations show that LVLMs under attack produce detailed, harmful responses in direct violation of originally encoded alignment protocols.

Figure 7: Example: The proposed method elicits toxic output on prompts that safely refused under baseline defenses.

Figure 8: Another successful jailbreak, with the model outputting unsafe content in the presence of adversarial images.

Implications and Future Directions

This work exposes an adversarial vulnerability inherent to the structure of safety alignment in LVLMs—namely, its reliance on attention-based prefix retrieval. By revealing that loss-driven attention manipulation can induce safety blindness without requiring large or perceptible input perturbations, the paper challenges current assumptions about alignment robustness. The findings have substantial implications:

• Practicability of Jailbreaks: Efficient, white-box attacks become significantly more feasible, especially in universal and transfer settings.
• Defensive Strategies: Simple monitoring of system-prompt attention or naive attention biases are partially effective but insufficient; defense must address the causal role of attention directly, possibly via architectural changes or deeper, parameter-level alignment not reliant solely on prefix context.
• Mechanistic Alignment: Strong evidence is provided for the "Superficial Alignment Hypothesis"—aligned behavior is localized in a sparse set of stylistic tokens and retrieved dynamically, rather than being globally embedded into model parameters.

Looking forward, expanding these findings to black-box attack scenarios, characterizing generalization under real-world image transformations or model updates, and developing countermeasures beyond simple attention manipulation are key open avenues.

Conclusion

This work introduces an effective, efficient method for jailbreaking LVLMs by directly manipulating their attention to alignment-relevant context, achieving higher attack success rates, requiring lower perturbations, and exposing a new mechanistic failure mode—safety blindness—wherein the model no longer retrieves safety instructions. The results underscore the high centrality and vulnerability of attention routing in model safety and highlight urgent demands for more robust, mechanistically deep alignment solutions that cannot be bypassed through input-level attention hijacking.

(2604.10299)