Vision-Language Models Create Cross-Modal Task Representations

Abstract: Autoregressive vision-LLMs (VLMs) can handle many tasks within a single model, yet the representations that enable this capability remain opaque. We find that VLMs align conceptually equivalent inputs into a shared task vector, which is invariant to modality (text, image) and format (examples, instruction), and may simplify VLM processing. We measure this alignment via cross-modal transfer -- the ability of a task vector derived in one modality to trigger the correct generation in another -- on a range of tasks and model architectures. Although the task vector is highly compressed, we find that this single vector outperforms prompting the model with the full task information, unique to this cross-modal case. Furthermore, we show that task vectors can be transferred from a base LLM to its fine-tuned vision-language counterpart, and that they can be derived solely from instructions without the need for examples. Taken together, our findings shed light on how VLMs internally process task information, and how they map different modalities into common semantic representations. Project page: https://vlm-cross-modal-reps.github.io.

• The paper demonstrates that VLMs encode tasks into shared embedding spaces, allowing effective transfer between text and image modalities.
• It reveals a consistent three-phase evolution in token representations—input, task, and answer—across modalities.
• Quantitative evaluations show that transferring text-derived task vectors to image queries significantly boosts accuracy and efficiency.

Overview of "Task Vectors are Cross-Modal"

The paper "Task Vectors are Cross-Modal" provides an in-depth investigation into the internal mechanisms of Vision-and-LLMs (VLMs), specifically focusing on how these models encode tasks. In essence, the authors explore how VLMs, which are capable of handling multi-modal inputs such as text and images, map various task specifications into a shared representation space termed "task vectors." This study reveals that tasks expressed through different modalities—whether as text examples, image examples, or instructions—are surprisingly encoded into similar task representations, enabling cross-modal transferability.

Key Findings

1. Cross-Modal Task Vectors: The research identifies that VLMs encode tasks into a shared embedding space that transcends the specific input modality. This implies that a task vector derived from one modality (e.g., text) can effectively guide the VLM when applied to a different modality (e.g., image), thereby facilitating cross-modal transfer. The authors demonstrate this with tasks involving mapping countries to capitals, animals to their scientific names, and so on.
2. Token Representation Phases: When processing inputs and generating responses, VLMs undergo a consistent evolution in their token representations across three distinct phases: input, task, and answer. This pattern holds irrespective of the input modality, highlighting a potentially universal mechanism in token representation evolution across VLM layers.
3. Transfer Performance: The authors quantitatively evaluate the performance of task vectors transferred across modalities. For instance, transferring text-derived task vectors to image queries improves accuracy compared to using unimodal image examples alone. Moreover, ensembling instruction-based vectors with exemplar-based vectors enhances the sample efficiency of task representation.
4. Inter-Model Transfer: A notable exploration within the work is the transferability of task vectors from pre-trained language-only models (LLMs) to fine-tuned VLMs. The study finds that task vectors in LLMs retain a high level of similarity with those in VLMs, allowing for effective cross-modal transfer from text-based queries processed by the LLM to image-based queries handled by the VLM.

Implications and Speculation

The insights from this paper have far-reaching implications for both theoretical understanding and practical applications in AI. The ability of VLMs to encode cross-modal, transferable task representations suggests that these models may be leveraging underlying commonalities between tasks across different modalities, potentially contributing to their generalist capabilities. This cross-modal task encoding could lead to more efficient AI systems that require fewer examples to generalize across different tasks and domains.

Theoretically, these findings challenge researchers to further explore the architectural and training methodologies that facilitate such cross-modal representations, potentially leading to breakthroughs in models of perception and cognition. Practically, the implications for AI development include the potential for creating more robust, efficient models capable of handling a wider range of inputs without exhaustive data requirements for each new task specification.

Conclusion

This paper contributes significantly to our understanding of multi-modal task processing in VLMs by unveiling the cross-modal nature of task vectors. By showing how task representations can transfer across modalities, the research opens new avenues for developing versatile multi-modal AI systems. Future work could investigate refining these models for better understanding diverse input contexts and task complexities, potentially leading to more integrated AI that mirrors human-like perception and problem-solving abilities.