Learning How To Ask: Cycle-Consistency Refines Prompts in Multimodal Foundation Models

Abstract: When LLMs perform zero-shot inference, they typically use a prompt with a task specification, and generate a completion. However, there is no work to explore the possibility of the reverse - going from completion to task specification. In this paper, we employ both directions to perform cycle-supervised learning entirely in-context. Our goal is to create a forward map f : X -> Y (e.g. image -> generated caption), coupled with a backward map g : Y -> X (e.g. caption -> generated image) to construct a cycle-consistency "loss" (formulated as an update to the prompt) to enforce g(f(X)) ~= X. The technique, called CyclePrompt, uses cycle-consistency as a free supervisory signal to iteratively craft the prompt. Importantly, CyclePrompt reinforces model performance without expensive fine-tuning, without training data, and without the complexity of external environments (e.g. compilers, APIs). We demonstrate CyclePrompt in two domains: code generation and image captioning. Our results on the HumanEval coding benchmark put us in first place on the leaderboard among models that do not rely on extra training data or usage of external environments, and third overall. Compared to the GPT4 baseline, we improve accuracy from 80.5% to 87.2%. In the vision-language space, we generate detailed image captions which outperform baseline zero-shot GPT4V captions, when tested against natural (VQAv2) and diagrammatic (FigureQA) visual question-answering benchmarks. To the best of our knowledge, this is the first use of self-supervised learning for prompting.

• The paper presents CyclePrompt, a novel cycle-consistency approach that refines multimodal prompts for enhanced task performance.
• It employs bidirectional mappings with a cycle-consistency loss to improve prompt quality without extra training data.
• Experiments demonstrate significant gains, with code generation accuracy rising from 80.5% to 87.2% and improved image captioning benchmarks.

Refining Prompts through Cycle-Consistency in Multimodal Foundation Models

Introduction

Prompt engineering is a critical aspect of working with LLMs, allowing for a range of applications from code generation to multimodal tasks like image captioning. Traditional approaches often rely on altering prompts to improve model performance, a necessity given the expensive nature of fine-tuning LLMs. This paper introduces an innovative method, named CyclePrompt, that iteratively refines prompts using a cycle-consistency mechanism, drawing on the concept of translating between modalities (e.g., text to code and image to text) and then back, to refine the initial prompt. Notably, this method achieves enhanced performance without the need for additional training data or complex external environments.

CyclePrompt Mechanism

The CyclePrompt methodology relies on establishing a forward and backward mapping between two data types (X and Y), such as code and text or images and captions. It formulates a cycle-consistency "loss" to refine prompts based on the discrepancy between the original input and the final output of these mappings. This approach resembles the autoencoder architecture but instead applies it for prompt refinement, employing the cycle as a supervisory signal. Significantly, CyclePrompt enhances model performance by refining the input prompt based on the cycle's outcome, thereby improving the quality of the model’s outputs in a self-supervised manner.

Experiments and Results

Code Generation

CyclePrompt was applied to the HumanEval benchmark for code generation, where it impressively reached first place among models not utilizing extra training data or external tools, with an accuracy improvement from 80.5% to 87.2% over the GPT4 baseline. This performance underlines the model's ability to refine its prompts for more precise code generation through iterative cycles.

Image Captioning

In the domain of image captioning, CyclePrompt was tested against natural and diagrammatic visual question-answering benchmarks. It achieved better performance in generating image captions compared to baseline zero-shot GPT4V captions. The cycle-consistency mechanism allowed the model to generate refined captions that led to improved question-answering abilities based on those captions alone.

Implications and Future Directions

The success of CyclePrompt in enhancing model performance through prompt refinement sans additional training data or complex external tools opens new avenues for research and application in the field of AI, particularly for code generation and image captioning tasks. It demonstrates the potential for cycle-based learning and refinement to improve the utility of LLMs across various domains, hinting at broader applications in enhancing model performance for other complex tasks.

The paper also lays groundwork for future explorations into the dynamics of forward and backward functions in cycle-consistency, the characterization of modality misalignment, and the potential for semantic gradient descent as a method for refining model understanding and performance iteratively. These areas present rich opportunities for advancing our understanding and leveraging of LLMs.

Conclusion

The introduction of CyclePrompt marks a significant step forward in the field of LLMs and generative AI, showcasing the power of cycle-consistency for prompt refinement without the need for additional resources. This method not only elevates the performance of LLMs in completing specific tasks but also broadens our understanding of effective interaction with these models, suggesting a promising direction for future research and development in AI.