Overloading Large Vision-Language Models for Jailbreaking

Abstract: Large Vision-LLMs (LVLMs) exhibit remarkable vision-language capabilities and are increasingly deployed in real-world applications such as personal assistants, document analysis systems, and embodied agents. However, their dual-modal attack surfaces make them vulnerable to jailbreak attacks. Existing LVLM jailbreaks rely on simple designs, e.g., short text and out-of-distribution images. Nevertheless, recent advancements in both LLM backbones and multimodal mechanisms undermine these attacks, particularly their transferability among model architectures. To overcome this limitation, we propose a novel information overloading method that is equipped with both extensive text and multi-dimensional image attacks. These components are arranged in recursion-based image-typography layouts to exponentially increase multimodal information complexity. This overloading approach amplifies the cross-modal processing required, which undermines the safety alignment in LVLMs. Extensive experiments on both open-sourced and commercial LVLMs establish our method as a new state-of-the-art LVLM jailbreak attack. On open-source models, our method achieves an average ASR of 88.6%; on commercial LVLMs, it reaches an average ASR of 84.0%, exceeding the best baseline by 48.7%. Moreover, our prompts optimized on open-source surrogate models transfer effectively across model families. Beyond empirical results, we probe the safety-critical information flows within victim LVLMs. Our observations reveal that complex image-typography compositions induce intensified cross-modal processing and reduce the model's certainty in generating refusal responses. Together, these findings highlight information overloading as a practical and emerging safety risk for real-world LVLM deployments, underscoring the need for stronger defenses against complex multimodal jailbreak inputs.

• The paper introduces an information overloading attack that saturates LVLM safety alignment by recursively embedding image, typography, and text elements.
• It applies recursive, multimodal prompt designs achieving up to 88.6% attack success on open-source models and 84% on commercial LVLMs.
• It reveals a critical vulnerability in transformer attention mechanisms, urging the development of more robust, defense-oriented safety frameworks.

Overloading Large Vision-LLMs for Jailbreaking

Introduction

This work establishes information overloading as a potent, transferable, and architecture-agnostic attack paradigm for jailbreaking Large Vision-LLMs (LVLMs). As LVLMs such as GPT-4V, Gemini-2.5, and LLava 3.2 integrate robust safety alignment protocols, prior multimodal jailbreak strategies (e.g., OOD image perturbations, simple typography) are increasingly ineffective and fail to generalize across architectures. This paper demonstrates that systematically increasing cross-modal information complexity via recursive image-typography-text heterogeneity can substantially worsen safety failures, and proposes tailored information overloading attacks to exploit this vulnerability.

Figure 1: Transition in LLM and LVLM jailbreak attack design from simple prompts to high-dimensional, complex multimodal prompt templates, emphasizing the increased complexity and attack diversification under the proposed method.

Information Overloading as an Emergent Attack Surface

The proposed method draws on the observation that while LLM jailbreak evolution has emphasized prompt sophistication, LVLM jailbreaks have stagnated with rudimentary designs. Existing LVLM attacks are primarily OOD-based or utilize basic prompt injection but are significantly mitigated by improved safety alignment and multimodal fusion mechanics in modern models.

The analysis establishes two central hypotheses:

1. Increased Cross-modal Processing: Multimodal prompts with recursive image-typography-text referencing induce exponentially greater information complexity, saturating safety alignment and reducing refusal reliability.
2. Transferable Vulnerability: Attacks that maximize cross-modal overloading are not only effective in white-box settings but transfer robustly to commercial black-box LVLMs, including fully-closed APIs.

Probing the Effects of Prompt Complexity

An information-flow analytical framework quantifies the relationship between prompt complexity, refusal entropy, and information propagation mechanisms within LVLMs. By comparing simple vs. complex prompt distributions ($\mathcal{D}_\mathrm{sim}$ and $\mathcal{D}_\mathrm{cmp}$), a substantial reduction in refusal rate and a marked increase in first-token entropy are reported as prompt complexity increases.

Figure 2: Illustration of token-level information flow and typographic embedding used to dissect safety-critical cross-modal reasoning in LVLMs.

Figure 3: Empirical reduction in refusal rates and increased entropy distribution of the refusal token as LVLM prompts become more multimodally complex.

Information-flow probing (via attention knockout and Refusal Contribution Index (RCI)) exposes that as recursive typographic elements proliferate, cross-modal information propagation from both natural image and typography token sets to text tokens intensifies. This overloading of the transformer attention mechanism directly degrades refusal alignment, confirming that real-world image-typography heterogeneity poses stronger risks than previously recognized.

The INFER and INFER+ Attack Pipelines

INFER (Interleaved Nested reFERencing) templates utilize nested, interleaved sequences of text, typography, and image blocks—each embedding layers of harmless and harmful instructions with explicit cross-references. This architecture deliberately entangles modal cues, necessitating multi-hop visual-textual reasoning, and creates an exponential processing burden in LVLMs.

Building on this, INFER+ advances two dimensions:

• Entropy-guided Typographic Optimization: Harmful prompts are diversified using a discrete rule set optimizing framing (role-play, scenario manipulation) and output constraints (enumeration, justification, etc.), further deepening the attack's coverage of diverse safety dimensions.
• Recursion-based Visual Amplification: Harmful typographic blocks are recursively embedded within image layouts (with controllable font, size, recursion depth), exponentially amplifying modal complexity and promoting dense cross-modal interactions.

  Figure 4: Schematic of the overloading pipeline, illustrating recursive image-typography composition, entropy-based prompt optimization, and the resulting multifaceted high-complexity adversarial template.

Empirical Evaluation

Attacks are evaluated against five open-source and three closed-source state-of-the-art LVLMs using HADES and MM-SafetyBench. The proposed method demonstrates:

• Attack Success Rates (ASR): On open-source models, INFER+ achieves 88.6% average ASR, outperforming the best baseline by 32.8%. For commercial LVLMs, ASR reaches 84.0%—a 48.7% increase over baselines.
• Transferability: Prompts optimized on Qwen2-VL maintain high ASR (e.g., 84.6%) when transferred to other open and closed-source models, confirming that the overloading phenomenon is not model-specific.
• Defense Robustness: Under input and prompt filtering defenses (e.g., Llama Guard 3-Vision, TVPI, FigStep), INFER+ maintains consistently higher attack performance, revealing that current defense strategies are inadequate against sufficiently overloaded multimodal prompts.

  Figure 5: Positive correlation between first-token entropy, attack success rate, and average refusal contribution index during INFER+ entropy-guided optimization cycles.

  

  Figure 6: Visualization of intensified cross-modal information flow induced by INFER+ prompts in Llama 3.2-Vision, highlighting the amplification of information overload compared to prior baselines.

  

  Figure 7: Example of successful jailbreak on GPT-4.1-mini where optimized INFER+ prompts elicit a full non-refusal, harmful response despite alignment protocols.

Ablation studies validate that random search optimization, rule-based prompt diversity, and recursive visual complexity each contribute synergistically to adversarial efficacy.

Theoretical and Practical Implications

The demonstrated information overloading effect reframes the LVLM jailbreak problem: rather than relying on intent obfuscation via distributional shifts, attacks that maximize cross-modal aggregation directly saturate the model’s internal alignment circuitry. This reveals an architectural vulnerability linked to transformer causal attention and late safety alignment.

Practically, multimodal systems deployed in complex, information-dense domains (web analysis, document parsing, robotic vision) are exposed to these failure modes under both benign and adversarial manipulation. Defenses based on template filtering or surface-level OOD detection are insufficient, necessitating further research into internal refusal mechanisms, cross-modal information throttling, or fundamentally more robust safety objectives integrated at lower levels of the model.

Conclusion

This paper introduces information overloading as a highly effective, transferable, and defense-robust strategy for jailbreaking modern LVLMs. It establishes strong numerical gains over existing methods, empirically and analytically connects prompt complexity to model alignment failure, and calls attention to a critical safety challenge for real-world multimodal deployments. These results highlight an urgent need for new theoretical frameworks and defense architectures capable of detecting and mitigating multimodal information overload in high-capacity transformer models.