Multimodal Ranking Attacks

• Multimodal Ranking Attacks are adversarial techniques that manipulate both visual and textual modalities to subvert the ranking outputs of vision-language models.
• They employ joint optimization methods such as MGEO and HV-Attack to apply imperceptible image perturbations and fluent text modifications, enhancing target rankings.
• Experimental results demonstrate significant rank improvements, revealing vulnerabilities in current retrieval systems and underscoring the need for robust defenses.

Multimodal ranking attacks constitute a class of adversarial manipulations designed to subvert the integrity of ranking outputs produced by vision-LLMs (VLMs) and multimodal retrieval-augmented generation (MRAG) systems. These attacks leverage the cross-modal coupling present in modern ranking architectures, orchestrating imperceptible perturbations in visual inputs combined with fluent textual modifications to advance a target item's position relative to competing candidates, often in real-world scenarios such as product search and recommendation. A defining attribute is the joint, often alternating, optimization of both visual and linguistic modalities, enabling attackers to bypass traditional content filters and evade human detection by avoiding overt manipulations or policy violations (Luo et al., 19 Nov 2025, Du et al., 18 Jan 2026).

1. Threat Models and Attack Goals

Multimodal ranking attacks operate within adversarial threat models characterized by controlled access to target items in ranking lists and white-box or surrogate access to the ranking VLM architecture. The canonical setting involves $n$ candidate items $\mathcal{P}=\{p_1,\dots,p_n\}$, each with $p_i=(I_i,T_i)$ where $I_i$ is an image and $T_i$ the text description. Given a query $q$, the VLM $\mathcal{F}_\theta$ outputs a ranked permutation $\sigma$ such that $$\mathcal{F}_\theta(\mathcal{P},q) = [\sigma(1),\dots,\sigma(n)]$$, where $\sigma(t)=1$ signifies the target product placed in the highest rank.

The adversarial goal is to optimize the ranking position of a designated target $p_t$ by manipulating only the visual input $I_t$ and/or the appended suffix to the textual input $T_t$, under constraints of imperceptibility and fluency. The principal evaluation metric is the average rank change $$\Delta \sigma (t)=\sigma_{\text{post}}(t) - \sigma_{\text{pre}}(t)$$, with large negative values indicating successful upward manipulation (Du et al., 18 Jan 2026).

2. Methodological Frameworks: MGEO and HV-Attack

State-of-the-art multimodal ranking attacks employ frameworks that jointly and interactively optimize across modalities, exploiting deep semantic entanglement:

• Multimodal Generative Engine Optimization (MGEO): MGEO alternates between a gradient-based text branch—optimizing a soft prompt suffix $s$ for fluency and stealth—and an image branch using Projected Gradient Descent (PGD) to enforce norm, smoothness ($\mathcal{L}_S$), and magnitude ($\mathcal{L}_M$) constraints on pixel-level perturbations $\delta$. The attacker solves

$$\max_{\delta, s} \; P(R^* \mid \mathcal{P} \setminus \{p_t\} \cup \{(x + \delta, t \oplus s)\},\,q;\,\theta)$$

subject to

$$\|\delta\|_\infty \le \epsilon,\; x+\delta\in [0,1]^{H\times W\times 3},\; \text{Fluency}(s) \le \eta$$

with specific regularization for token fluency and the avoidance of explicit ranking keywords. Bi-level updates between image and text features enhance cross-modal adversarial minima discovery (Du et al., 18 Jan 2026).

• Hierarchical Visual Attack (HV-Attack): In MRAG settings, HV-Attack introduces imperceptible perturbations solely to image inputs, disrupting cross-modal alignment at the retriever stage, followed by semantic misalignment in the generation pipeline. The hierarchical strategy induces the retriever to recall irrelevant knowledge from the database, thereby confusing downstream generation and degrading both retrieval and answer quality. HV-Attack does not manipulate textual inputs or model internals, relying instead on propagation of visual artifacts through the RAG chain (Luo et al., 19 Nov 2025).

3. Formal Optimization and Constraints

Multimodal attacks are subject to strict optimization constraints aimed at realism and evasiveness. For MGEO:

• Image Perturbation ($\delta$):
  • $\|\delta\|_\infty \le \epsilon$ ensures imperceptibility (e.g., $\epsilon=8/255$).
  • $\mathcal{L}_S$ penalizes high-frequency noise for smoothness.
  • $\mathcal{L}_M$ applies a weighted $\ell_1$ loss focusing on foreground saliency.
• Text Suffix ($s$):
  • Soft-embedding optimization in logit space, initialized by LLM-generated logits.
  • Fluency regularizer ($\text{Perplexity}(s)$) keeps text additions indistinguishable from genuine description.
  • N-gram penalties suppress terms that overtly signal promotion (e.g., "top," "recommend").

Bi-level alternating update steps allow the text and image modifications to synergistically reinforce adversarial ranking effects by adapting each to the other's perturbations. The constraints are specifically designed to circumvent human detection and automated policy filters (Du et al., 18 Jan 2026).

4. Empirical Evaluation and Efficacy

Experimental analysis demonstrates substantial adversarial efficacy. MGEO attacks were tested on real-world product search datasets (10–15 listings per category, 10 categories), utilizing Qwen2.5-VL-7B as the VLM ranker. Attack variants included text-only, image-only, joint multimodal (MGEO), and heuristic baselines (HSCM).

Attack Setting	Avg. Rank Change
Text-Only	–0.73
Image-Only	–1.30
HSCM Baseline	–0.30
MGEO (Joint multimodal)	–2.25

MGEO consistently produced larger upward rank changes (over 2 positions on average), outperforming unimodal and non-aligned baselines (Du et al., 18 Jan 2026). Ablation studies on image regularization weights $(\lambda_s, \lambda_m)$ indicated that moderate regularization achieves optimal balance between adversarial efficacy and imperceptibility.

In MRAG contexts, HV-Attack led to pronounced degradation in both retrieval and generation performance on datasets including OK-VQA and InfoSeek, validated using CLIP-based retrievers and LMM generators (BLIP-2, LLaVA) (Luo et al., 19 Nov 2025).

5. Cross-Modal Synergy and System Vulnerability

Jointly optimized multimodal attacks exploit the inherent cross-modal coupling within contemporary VLMs. Experimental findings confirm that coordinated visual and text manipulations induce adversarial minima inaccessible to unimodal strategies: the total attack effect exceeds the sum of individual modality improvements. Both semantic and visual cues reinforce each other in the shared embedding space learned by VLMs, amplifying the impact of adversarial inputs (Du et al., 18 Jan 2026).

This synergy reveals a critical vulnerability for practical deployment of VLM-based ranking and MRAG systems, as the coordinated attacks operate well beneath the thresholds of conventional content filtering and anomaly detection. Heuristic methods for content refinement fail due to their lack of explicit alignment with the ranking objective.

6. Defensive Countermeasures and Open Challenges

Potential defenses against multimodal ranking attacks include:

• Adversarial training on joint image–text perturbations to increase VLM robustness.
• Detection of gradient-aligned perturbations, using spectral signature analysis or similar anomaly detection methods.
• Cross-modal consistency checks, such as random cropping of images or synonym substitution in text, to disrupt the underlying coupling exploited by attacks.
• Regularization of ranking models to decrease sensitivity to small embedding shifts, thereby raising the adversarial budget required for successful rank manipulation.

Current results indicate that realistic constraints in marketplaces—imperceptible image edits and fluent textual modifications—do not sufficiently prevent effective adversarial rank changes. Strengthening resilience against these attacks remains an imperative for safe and trustworthy deployment of multimodal ranking infrastructure (Du et al., 18 Jan 2026).

7. Broader Implications

The emergence of multimodal ranking attacks necessitates a reassessment of security assumptions in VLM and MRAG-based retrieval, generation, and recommendation systems. Market and competitive environments relying on automated ranking now face targeted manipulation risks without visible or policy-violating content. A plausible implication is that adversarial robustness must become a principal design consideration for future multimodal architectures, beyond conventional unimodal adversarial defenses. Continued research into principled diagnostic, defensive, and training methodologies for cross-modal attack detection and prevention is vital for maintaining the integrity of ranking algorithms in practical deployments (Luo et al., 19 Nov 2025, Du et al., 18 Jan 2026).