Enhance Reasoning Ability of Visual-Language Models via Large Language Models

Abstract: Pre-trained visual LLMs (VLM) have shown excellent performance in image caption tasks. However, it sometimes shows insufficient reasoning ability. In contrast, LLMs emerge with powerful reasoning capabilities. Therefore, we propose a method called TReE, which transfers the reasoning ability of a LLM to a visual LLM in zero-shot scenarios. TReE contains three stages: observation, thinking, and re-thinking. Observation stage indicates that VLM obtains the overall information of the relative image. Thinking stage combines the image information and task description as the prompt of the LLM, inference with the rationals. Re-Thinking stage learns from rationale and then inference the final result through VLM.