YoChameleon: Personalized Vision and Language Generation

Abstract: Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into powerful tools with millions of users. However, they remain generic models and lack personalized knowledge of specific user concepts. Previous work has explored personalization for text generation, yet it remains unclear how these methods can be adapted to new modalities, such as image generation. In this paper, we introduce Yo'Chameleon, the first attempt to study personalization for large multimodal models. Given 3-5 images of a particular concept, Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information to (i) answer questions about the subject and (ii) recreate pixel-level details to produce images of the subject in new contexts. Yo'Chameleon is trained with (i) a self-prompting optimization mechanism to balance performance across multiple modalities, and (ii) a ``soft-positive" image generation approach to enhance image quality in a few-shot setting.

Overview of Yo'Chameleon: Personalized Vision and Language Generation

The paper "Yo'Chameleon: Personalized Vision and Language Generation" introduces a novel approach to personalization within Large Multimodal Models (LMMs). These models, like Chameleon, are capable of processing and generating both visual and textual information, serving as versatile AI assistants. While they excel at general tasks, they often lack the capacity for personalized query handling, which is crucial for meaningful human-AI interactions. The paper proposes a method to address this gap by incorporating user-specific concepts into LMMs.

Key Contributions and Methodology

The authors present Yo'Chameleon, a groundbreaking framework for personalizing LMMs. The personalization is achieved by using a small dataset of 3-5 images to represent a specific concept, leveraging soft-prompt tuning to embed subject-specific information. The framework features two prominent innovations:

1. Soft-Positive Image Utilization: The study introduces a "soft-positive" image generation method to enhance image quality. By utilizing visually similar images to augment training data, the model can create high-quality personalized outputs even with a few samples.

2. Dual Prompt and Self-Prompting Mechanism: Recognizing the distinct requirements for text understanding and image generation, the authors propose a dual soft prompt strategy. Additionally, a self-prompting mechanism is introduced, allowing dynamic token selection based on the task, enhancing the model's personalized capabilities across modalities.

Results and Implications

Quantitative and qualitative evaluations indicate that Yo'Chameleon surpasses baseline models in efficiently learning and generating personalized content. It achieves superior recognition accuracy and improved image generation quality, validating its effectiveness in embedding personalized knowledge into multimodal models without degrading their general capabilities.

With these advancements, Yo'Chameleon opens new avenues in AI personalization, potentially transforming LMMs from generic assistants to personalized AI entities capable of context-specific interactions. This progress could have significant implications for developing more intuitive and responsive AI systems across various applications, including virtual assistants, personalized content generation, and customer service.

Future Directions

The approach set forth in Yo'Chameleon paves the way for further exploration into enhancing personalization across different modalities. Future research could focus on increasing the number of personalized concepts, improving the representation of complex subjects, and addressing challenges in personalizing dynamic entities such as body movements or voices. As the AI landscape continually evolves, methodologies like those proposed in Yo'Chameleon could play a critical role in shaping adaptive, user-centered models capable of bridging the gap between generic AI functions and personalized user experiences.