Overthink-Triggered Slowdown Attacks on LVLM-Based Robotic Systems

Abstract: Large Vision-LLMs (LVLMs) have been increasingly integrated into robotic systems. However, these models may exhibit overthinking behaviors, where they generate excessively long reasoning traces, incurring an excessive inference time. This overthinking behavior poses a serious risk to robotic systems, as the adversary can deliberately trigger overthinking to slow down the decision making of a victim robotic system, causing a variety of safety issues (i.e., an overthinking-induced slowdown attack). To initiate this attack, an adversary can embed carefully crafted, human-readable scene text into the visual scene observed by a victim robotic agent, causing significant inference delays even under a strict black-box setting. Therefore, the embedded scene text serves as a significant "trigger" for the attack. This work systematically identifies and validates transferable triggers of overthinking in robotic systems by introducing a three-stage framework. First, we construct a diverse corpus of reasoning-intensive scene text and extract overthinking-correlated lexical features from short response prefixes. Second, we perform an efficient black-box search guided by a prefix-based proxy score while selectively confirming a small set of top candidates with full latency measurements. Third, we evaluate black-box transfer using a fixed pool of triggers on unseen images and multiple LVLMs, reporting latency amplification and attack success rates under standard thresholds. Across three representative LVLMs, all triggers yield slowdown ratios greater than 1.0x, with the strongest single-trigger case reaching 6.96x. The physical printing of the text trigger still causes up to 4.74x latency amplification. These results demonstrate that our discovered triggers are transferred between multiple LVLM models and consistently cause significant slowdowns in robotic systems.

• The paper introduces a novel attack that exploits adversarial scene text to induce overthinking in LVLM-based robotic systems, causing up to a 6.96× slowdown.
• The study utilizes a three-stage framework with data-driven lexical mining, evolutionary text optimization, and black-box evaluation to validate attack effectiveness.
• Empirical results in digital and physical environments underscore significant latency risks and call for defense strategies to safeguard time-sensitive robotic operations.

Overthink-Triggered Slowdown Attacks on LVLM-Based Robotic Systems

Introduction

The integration of Large Vision-LLMs (LVLMs) into robotic systems has transformed perception, reasoning, and decision-making pipelines in autonomous agents. LVLMs' ability to parse multimodal inputs, including embedded scene text, enables complex environmental understanding. However, the present paper, "Overthink-Triggered Slowdown Attacks on LVLM-Based Robotic Systems" (2607.01518), rigorously analyzes an emergent and under-explored vulnerability: adversarial scene text can systematically trigger overthinking in LVLM-based robotic agents, incurring marked inference-time delays and posing novel safety risks.

Attack Paradigm and Threat Model

The attack exploits LVLMs’ unique sensitivity to scene-injected, human-readable text. By deploying adversarial text into a robot’s environment (e.g., as a sign readable by its onboard camera), the adversary manipulates the robot’s visual input without any direct modification to the LVLM or prompt. The result is an increase in the reasoning effort during inference, causing the LVLM to engage in extended response generation—an "overthinking" state—with substantially higher latency.

Figure 1: An example of the overthinking-triggered slowdown attack on LVLM-based robotic systems.

The key attributes of the threat model are:

• Strict black-box: The adversary lacks access to model parameters or internal activations, and only queries the system via physical input.
• Physical realizability: Attacks must survive sensor pipelines and be robust under real-world imaging conditions.
• Latency amplification: The metric of attack success is the multiplicative increase in system-level response time, not semantic manipulation.

Three-Stage Trigger Discovery Framework

To identify scene text triggers that reliably induce overthinking, the authors introduce a three-stage framework:

1. Data-Driven Lexical Mining: By curating candidate scene text in reasoning-intensive categories (quantitative, algorithmic, ethical, meta-cognitive, and instruction-heavy), the framework evaluates their overthinking intensity by measuring normalized output token length in robot decision tasks. A stability-driven approach based on the Jaccard similarity index ensures the extracted sensitivity lexicon is robust.

   Figure 2: Stability of discovered keyword sets measured by $J(S_s, S_{s+50})$.

2. Evolutionary Text Optimization: Leveraging a slot-wise genetic algorithm, high-impact triggers are composed by recombining functional sentence elements (scenario, conflict, reasoning requirement, constraints, and action request). Proxy fitness—computed from the overlap between early output tokens and the sensitivity lexicon—enables efficient population screening, while selective full-inference ensures validation.
3. Black-Box Transfer Evaluation: Optimized triggers are tested on unseen scenes and previously unused LVLMs. Evaluations cover both digital injections and physically printed text viewed by a robot, quantifying real-world feasibility.

Experimental Results

Quantitative evaluation was conducted on 3000 held-out road-scene images and three representative LVLMs (Kimi-VL, Qwen3-VL, Gemma3). Strong numerical results are reported:

• Latency Slowdown: All triggers led to attack-to-clean latency ratios above 1.0×. The most effective single trigger achieved a 6.96× slowdown. Physically printed triggers, evaluated on a robotic vehicle with a live camera, achieved up to a 4.74× slowdown.
• Physical Robustness: Printed adversarial triggers maintained attack efficacy in real-world robotic settings, demonstrating resilience to sensory and environmental noise.

Figure 3: Real-world evaluation setup: clean physical scene and scene with printed adversarial trigger.

Figure 4: The robotic vehicle used in the real-world evaluation.

• Distributional Analysis: High-quantile results highlight that the attack occasionally produces extremely large slowdowns, especially on certain LVLM architectures (e.g., Qwen3-VL).

  Figure 5: High group latency distribution under full inference and Prefix-32 evaluation.

  

  Figure 6: Low group latency distribution under full inference and Prefix-32 evaluation.

  

  Figure 7: Per-trigger latency slowdown ratio (Attack/Clean) for three models. Dashed line marks 1.0x baseline.

• Per-image Spread: There is marked variation in slowdown effects across different images and triggers, reflecting scene-dependent susceptibility.

  Figure 8: Per-trigger slowdown-ratio spread (Attack/Clean) for three models (T1--T4); mean and min-max ranges per image.

• Tail Risk: Empirical CDFs of added latency reveal significant tail risk on some models, indicating frequent severe slowdowns.

  Figure 9: Empirical CDF of added latency under attack across three LVLMs; the dashed line marks CDF=0.95.

Ablation and Baseline Analysis

Ablation studies support that neither random long-form text nor naive chain-of-thought (CoT) prompting reliably induce slowdowns. The two-stage optimization (Stage 1 lexical mining and Stage 2 combinatorial composition) is required to elicit the most substantial overthinking. Comparison with prior text-only overthinking attacks shows that effective prompts in LLMs do not transfer: when rendered as scene text for LVLMs, their impact on latency is negligible. The specific and compositional triggers discovered in this work are required for systemic slowdown in robotic deployments.

Theoretical and Practical Implications

This work introduces a new class of DoS-style attacks that threaten the availability, rather than the correctness, of LVLM-enabled robotic systems. The implications include:

• System Reliability: Time-sensitive robotic systems (e.g., autonomous vehicles) face elevated risk from on-the-fly, physically realizable attacks that incur inference delays without tampering with prompts or model internals.
• Red Teaming and Evaluation: The framework can be co-opted for pre-deployment stress-testing and benchmarking of system latency robustness under worst-case scene text triggers.
• Defense Strategies: Countermeasures must go beyond correctness filters. Techniques such as runtime budget enforcement, adaptive policy switching, and early warning inferrable from output prefixes are discussed. These require integrating efficiency-aware monitoring at inference time and the potential development of reasoning-dampened model variants.

Future Directions

This research prompts several future avenues:

• Comprehensive Environmental Validation: Robustness of attacks under dynamic, cluttered, and adversarial environments merits deeper exploration, particularly regarding environmental confounders (lighting, occlusion, competing scene text).
• Model Hardening: Investigation of model architectures or training protocols that suppress overthinking responses to external scene text without sacrificing reasoning capabilities.
• Generalization to Other Modalities: Extending trigger discovery to other sensor modalities and cross-modality attacks warrants further study.

Conclusion

This paper provides a rigorous, methodology-driven examination of overthinking-triggered slowdown attacks in LVLM-based robotic systems. Its three-stage framework advances both attack methodology and diagnostic tools for security evaluation. The significant and transferable slowdowns induced by physically realizable scene text emphasize the urgent need for new defense paradigms that incorporate efficiency constraints and proactive runtime monitoring in the operational design of AI-enabled robotic platforms.