Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?

Abstract: Recently, newly developed Vision-LLMs (VLMs), such as OpenAI's o1, have emerged, seemingly demonstrating advanced reasoning capabilities across text and image modalities. However, the depth of these advances in language-guided perception and abstract reasoning remains underexplored, and it is unclear whether these models can truly live up to their ambitious promises. To assess the progress and identify shortcomings, we enter the wonderland of Bongard problems, a set of classic visual reasoning puzzles that require human-like abilities of pattern recognition and abstract reasoning. With our extensive evaluation setup, we show that while VLMs occasionally succeed in identifying discriminative concepts and solving some of the problems, they frequently falter. Surprisingly, even elementary concepts that may seem trivial to humans, such as simple spirals, pose significant challenges. Moreover, when explicitly asked to recognize ground truth concepts, they continue to falter, suggesting not only a lack of understanding of these elementary visual concepts but also an inability to generalize to unseen concepts. We compare the results of VLMs to human performance and observe that a significant gap remains between human visual reasoning capabilities and machine cognition.

• The paper demonstrates that state-of-the-art Vision-Language Models, like GPT-4o, struggle with abstract visual reasoning, solving only 21 out of 100 Bongard puzzles.
• The research employs diverse evaluation formats, with Claude scoring 28 correct answers using multiple-choice rule pairs and 69 correct answers when choices were reduced.
• The findings underscore fundamental perceptual challenges in VLMs and advocate for advanced techniques like contrastive learning to enhance image encoding and reasoning abilities.

Overview of "Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?"

This paper presents an empirical study on the capabilities of Vision-LLMs (VLMs) when faced with Bongard problems (BPs), a set of visual reasoning puzzles that test pattern recognition and abstract reasoning. The authors address the gap in understanding the reasoning capabilities of VLMs, noting that despite advancements, these models still struggle with visual cognition tasks that are trivial for humans.

Evaluation of Vision-LLMs

The authors evaluate several state-of-the-art VLMs, including GPT-4o, Claude, Gemini, and LLaVA, using a dataset of 100 original Bongard problems. They also compare these results with human performance, highlighting significant disparities in understanding visual concepts.

Key Findings:

• VLMs showed limited success, with GPT-4o solving 21 out of 100 problems, highlighting a considerable gap between machine and human cognitive abilities.
• When models were provided with multiple-choice rule pairs, Claude performed slightly better, solving 28 problems.
• Reducing the complexity to offer only 10 possible solution choices improved performance, with Claude solving 69 problems.

Analysis of Concepts and Limitations

The study explores specific Bongard problems and reveals that misconstrued concepts primarily drive the models’ failures. In cases like BP#16 and BP#55, VLMs struggled with basic visual concepts of spirals and spatial positioning, respectively. Only with relatively simple tasks, such as identifying shapes in BP#36, did the models show a better comprehension level.

Implications and Future Directions

These findings underscore the limitations of VLMs in visual reasoning and suggest that while models can mimic certain aspects of human reasoning, fundamental perceptual challenges remain. The implications highlight the need for targeted improvements in image encoding and reasoning capabilities.

The research proposes further investigation into the architectures' latent spaces and suggests leveraging advanced techniques such as contrastive learning or program synthesis for enhanced concept linking and visualization capabilities.

Conclusion

The authors conclude that while VLMs show promise in specific domains, a significant gap persists in abstract visual reasoning. Continued focus on cognitive benchmarks, perceptual accuracy, and innovative methodologies could bridge this divide, advancing the field of AI towards genuine human-like reasoning.