ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

Abstract: Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences. While such models can provide a powerful score for matching and subsequent zero-shot tasks, they are not capable of generating caption given an image. In this work, we repurpose such models to generate a descriptive text given an image at inference time, without any further training or tuning steps. This is done by combining the visual-semantic model with a LLM, benefiting from the knowledge in both web-scale models. The resulting captions are much less restrictive than those obtained by supervised captioning methods. Moreover, as a zero-shot learning method, it is extremely flexible and we demonstrate its ability to perform image arithmetic in which the inputs can be either images or text, and the output is a sentence. This enables novel high-level vision capabilities such as comparing two images or solving visual analogy tests. Our code is available at: https://github.com/YoadTew/zero-shot-image-to-text.

• The paper introduces ZeroCap, achieving zero-shot image-to-text generation by combining CLIP for image-text alignment with GPT-2 for language generation.
• It employs a CLIP-based loss to guide GPT-2 during inference, balancing semantic accuracy with linguistic fluency through gradient descent optimization.
• Empirical results demonstrate strong semantic alignment and novel vocabulary generation, highlighting ZeroCap's potential for versatile multi-modal reasoning.

ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

The paper presents ZeroCap, an innovative approach enabling zero-shot image captioning by leveraging a combination of the CLIP and GPT-2 models. Unlike conventional supervised captioning methods, ZeroCap achieves this task without additional training, providing a flexible solution for visual-semantic tasks by repurposing existing large-scale models.

Overview

ZeroCap integrates two powerful pre-trained models: CLIP, for image-text alignment, and GPT-2, for language generation. This architecture enables generating descriptive text for a given image purely based on inference. The methodology capitalizes on CLIP's ability to effectively match images with textual descriptions and GPT-2's text generation capabilities, circumventing the limitations of curated datasets typical in supervised learning. The result is a novel capacity for performing tasks such as visual-semantic arithmetic, which broadens the spectrum of what zero-shot learning can achieve.

Technical Approach

The approach involves guiding GPT-2 with a CLIP-based loss during inference. The optimization modifies the context as tokens are generated, adjusting word probabilities to reflect the image content more accurately. The paper outlines the optimization challenge as a balance between aligning generated text with the image and maintaining linguistic properties, employing gradient descent to adjust the cache during token generation. The framework goes beyond traditional captioning by allowing arithmetical operations within the semantic vector space, using both images and text to derive meaningful relational descriptions.

Results and Comparisons

Empirical results highlight the distinctiveness of ZeroCap's outputs compared to supervised baselines. While traditional supervised metrics such as BLEU and CIDEr show lower scores for ZeroCap, reflecting a departure from human-annotated labels, unsupervised evaluations demonstrate strong semantic alignment with images through CLIP-Score metrics. Furthermore, ZeroCap generates more diverse and novel vocabulary, evidencing its capacity for creative, context-driven descriptions. The visual-semantic arithmetic capabilities expand the potential applications of this methodology, enabling zero-shot solutions for tasks involving relational reasoning and analogy-solving.

Implications and Future Prospects

ZeroCap marks an important milestone in leveraging pre-trained models for generative tasks within computer vision. The paper suggests new directions for research involving the integration of AI systems capable of multi-modal reasoning without the necessity for exhaustive re-training, highlighting the potential for scaling such approaches in diverse domains. The study provides a framework for exploring more intricate visual-textual interactions, potentially revolutionizing domains like automated video summarization, context-aware robotics, and advanced multimedia retrieval systems.

In summary, ZeroCap exemplifies an effective strategy for zero-shot image-to-text generation by combining the strengths of large-scale language and vision models, illustrating the transformative potential of integrated AI solutions. The implications of this study extend into future developments in AI, enabling robust, flexible, and scalable systems capable of complex understanding and generation tasks across various fields.