IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidance

Abstract: Many Vision-Language-Action (VLA) models flatten image patches into a 1D token sequence, weakening the 2D spatial cues needed for precise manipulation. We introduce IVRA, a lightweight, training-free method that improves spatial understanding by exploiting affinity hints already available in the model's built-in vision encoder, without requiring any external encoder or retraining. IVRA selectively injects these affinity signals into a language-model layer in which instance-level features reside. This inference-time intervention realigns visual-token interactions and better preserves geometric structure while keeping all model parameters fixed. We demonstrate the generality of IVRA by applying it to diverse VLA architectures (LLaRA, OpenVLA, and FLOWER) across simulated benchmarks spanning both 2D and 3D manipulation (VIMA and LIBERO) and on various real-robot tasks. On 2D VIMA, IVRA improves average success by +4.2% over the baseline LLaRA in a low-data regime. On 3D LIBERO, it yields consistent gains over the OpenVLA and FLOWER baselines, including improvements when baseline accuracy is near saturation (96.3% to 97.1%). All code and models will be released publicly. Visualizations are available at: jongwoopark7978.github.io/IVRA

• The paper introduces IVRA, a training-free module that re-injects instance-level spatial affinity into language models to enhance robot action policies.
• The methodology computes a normalized affinity matrix from frozen vision encoder features to dynamically mix visual tokens while preserving local structure.
• Experimental results demonstrate significant improvements in both simulated benchmarks and real-world tasks, leading to more precise and robust robot manipulation.

Training-Free Affinity-Guided Visual Token Mixing for Precise Robot Manipulation

Introduction

Vision-Language-Action (VLA) models are central to robotic manipulation via multimodal policy generation, especially when tasks require precise interpretation of visual contexts and natural language instructions. The standard approach for assembling image-text inputs in VLA models is to flatten image patch tokens into 1D sequences for ingestion by Transformers. This strategy, while architecturally convenient, erodes spatial correlations and instance-level visual cues, impairing the downstream robot policy’s ability to segment, locate, and interact with objects at fine granularity. The paper "IVRA: Improving Visual-Token Relations for Robot Action Policy with Training-Free Hint-Based Guidance" (2601.16207) introduces IVRA, a lightweight, training-free, inference-time module to mitigate this loss by re-injecting instance-level spatial affinity directly into selected LLM layers.

Figure 1: IVRA overview: frozen vision encoder extracts affinity hints for visual token mixing to recover instance-level structure and object boundaries essential for robot manipulation.

Methodology

IVRA capitalizes on internal correlation (affinity) signals already embedded within the frozen vision encoder of VLA models (e.g., CLIP, DINO). For each input image, the encoder yields intermediate patchwise features $\{f_i\}$. IVRA computes a normalized affinity matrix $A \in \mathbb{R}^{N \times N}$ via cosine similarity across all patch pairs:

$A_{ij} = \frac{f_i \cdot f_j}{\|f_i\| \|f_j\|}$

High affinity denotes visual similarity and local co-occurrence indicative of shared object membership.

Within the LLM pipeline, visual tokens replacing the <image> placeholder are dynamically pooled at selected layers using affinity weights. Specifically, each token $v_i$ at layer $l$ is updated as:

$$v'_i = \sum_j \alpha_{ij} v_j, \qquad \alpha_{ij} = \frac{\max(A_{ij},0)}{\sum_k \max(A_{ik},0)}$$

The final token is linearly mixed with its original state via a convex combination ($\lambda$ empirically chosen as 0.3):

$v_i^{\text{mix}} = (1-\lambda) v_i + \lambda v'_i$

This operation preserves global semantics while locally enhancing instance-level boundaries. Visual tokens only are affected; text tokens are never modified.

IVRA is integrated into a single or few layers near the top of the LLM, with ablation revealing that early injection (layer 20) and moderate mixing optimize performance without incurring significant computational overhead (3% latency increase, zero parameter growth).

Experimental Results

Simulated Benchmarks

VIMA Tasks (2D Manipulation):

Application of IVRA to LLaRA achieves significant improvement in average success rate over vanilla LLaRA in low-data regimes (12% of expert trajectories). The margin is strongest for tasks with novel object combinations and novel attribute generalization. For instance:

• Novel Object: +4.2% (57.1%→61.3%)
• Object Place: +3.5% (69.6%→73.1%) This demonstrates robust generalization, with IVRA outperforming even methods using oracle detectors and full data.

LIBERO Suite (3D Manipulation):

When injected into OpenVLA and FLOWER models, IVRA consistently yields positive gains:

• FLOWER-90: +2.6% (93.4%→96.0%)
• FLOWER-Object: +0.6% (99.3%→99.9%) Critically, improvements are observed even when baseline accuracy is near saturation (e.g., from 96.3% to 97.1%), confirming IVRA's complementary instance-level structuring rather than compensation for weak baselines.

Real-World Robot Tasks

A zero-shot real robot setup was evaluated using models trained exclusively on synthetic data. Four distinct real-world tasks require identifying objects either by explicit type, color match, localization in clutter, or physical attribute (height).

Figure 2: Hardware setup: robot arm with gripper and overhead RGB camera providing workspace observation.

In all four tasks, LLaRA+IVRA significantly surpasses LLaRA:

• Target Object: +10% (50%→60%)
• Color Match: +30% (30%→60%)
• Cluttered Localization: +30% (45%→75%)
• Object Height: +20% (50%→70%)

  Figure 3: Representative real-world scenarios—initial scene (top) and successful end state (bottom) with distracting objects present.

Task-specific analysis in Figures 4–7 shows IVRA’s effect:

• In T1, both LLaRA and LLaRA+IVRA succeed, indicating trivial objects are already separable.
• In T2, LLaRA+IVRA achieves correct color-based selection, whereas the baseline fails.
• For T3, IVRA robustly localizes the intended target in clutter, in contrast to baseline misselections.
• In T4, IVRA selects the correct length-based object, whereas the baseline confuses spatial attributes.

Figure 4: T1—target orange picked and dropped correctly by both methods.

Figure 5: T2—only LLaRA+IVRA picks broccoli matching duck’s color; baseline fails.

Figure 6: T3—LLaRA+IVRA correctly picks yellow duck in clutter; baseline picks wrong object.

Figure 7: T4—LLaRA+IVRA accurately picks the long corn; baseline picks short strawberry.

Qualitative Analysis

Affinity map visualizations reveal clear denoising effects under IVRA compared to baseline VLA pipelines. Object boundaries are sharper, activations strongly localized, and cross-object feature bleeding is notably reduced after IVRA application. This is consistent across both synthetic and real image domains. Enhanced maps support more robust reasoning about object attributes and spatial relations, as required for manipulative actions.

Theoretical and Practical Implications

IVRA’s inference-time, training-free strategy sidesteps the need for large-scale retraining, external modules, or region annotations. This marks a departure from methods relying on region-level grounding (RegionCLIP, Grounding~DINO) or open-vocabulary detection, which typically demand substantial architectural or data overhead. Instead, IVRA directly leverages built-in feature-level spatial correlations, making it broadly applicable to existing VLA architectures.

Practically, IVRA’s gains in zero-shot real-world tasks directly translate to more reliable and precise robot manipulation, particularly in low-data or domain adaptation setups. Theoretically, IVRA highlights the criticality of retaining mid-level instance-level structuring during model fusion, as opposed to naive global flattening. Future developments might evaluate dynamic affinity integration, hierarchical token mixing, or cross-modal joint affinity modeling to further enhance fine-grained reasoning for embodied agents.

Conclusion

IVRA presents a targeted, efficient solution to the spatial dilution inherent in current VLA pipelines. By injecting vision encoder affinity hints into selected LLM layers at inference—without retraining—the method consistently enhances instance-level recognition, localization, and physical reasoning, improving both manipulation success rates in simulation and real-world generalization. This work sets a foundation for the progressive integration of vision-language spatial priors in multimodal robotics and suggests promising directions for future policy composition and spatial reasoning strategies in AI-embodied agents.