ChatGPT-Powered Hierarchical Comparisons for Image Classification

Abstract: The zero-shot open-vocabulary challenge in image classification is tackled by pretrained vision-LLMs like CLIP, which benefit from incorporating class-specific knowledge from LLMs like ChatGPT. However, biases in CLIP lead to similar descriptions for distinct but related classes, prompting our novel image classification framework via hierarchical comparisons: using LLMs to recursively group classes into hierarchies and classifying images by comparing image-text embeddings at each hierarchy level, resulting in an intuitive, effective, and explainable approach.