Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search

Abstract: In this work, we aim to develop an MLLM that understands and solves questions by learning to create each intermediate step of the reasoning involved till the final answer. To this end, we propose Collective Monte Carlo Tree Search (CoMCTS), a new learning-to-reason method for MLLMs, which introduces the concept of collective learning into ``tree search'' for effective and efficient reasoning-path searching and learning. The core idea of CoMCTS is to leverage collective knowledge from multiple models to collaboratively conjecture, search and identify effective reasoning paths toward correct answers via four iterative operations including Expansion, Simulation and Error Positioning, Backpropagation, and Selection. Using CoMCTS, we construct Mulberry-260k, a multimodal dataset with a tree of rich, explicit and well-defined reasoning nodes for each question. With Mulberry-260k, we perform collective SFT to train our model, Mulberry, a series of MLLMs with o1-like step-by-step Reasoning and Reflection capabilities. Extensive experiments demonstrate the superiority of our proposed methods on various benchmarks. Code will be available at https://github.com/HJYao00/Mulberry

• The paper introduces CoMCTS, a novel collective learning framework that integrates knowledge from multiple MLLMs for effective reasoning.
• It leverages iterative operations—Expansion, Simulation, Backpropagation, and Selection—to construct efficient reasoning paths using the Mulberry-260k dataset.
• The Mulberry models demonstrate enhanced reflective reasoning and computational efficiency, outperforming many open-source alternatives.

Overview of "Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search"

The paper presents a novel approach called Collective Monte Carlo Tree Search (CoMCTS), aimed at enhancing the reasoning capabilities of multimodal LLMs (MLLMs). This method introduces the concept of collective learning into the field of tree search strategies, traditionally dominated by self-bootstrapping approaches. By leveraging a collective framework, CoMCTS significantly improves both the effectiveness and efficiency of reasoning-path searching and learning tasks in MLLMs.

Key Contributions

The crux of CoMCTS lies in its ability to integrate collective knowledge from multiple MLLMs, thereby addressing the shortcomings of traditional methods like MCTS that often dwell in homogeneous low-quality nodes. The CoMCTS paradigm facilitates effective reasoning by involving four iterative operations—Expansion, Simulation and Error Positioning, Backpropagation, and Selection. Each of these operations harnesses knowledge from various models to expand, simulate, and optimize reasoning paths collectively.

1. Integration of Collective Learning into MCTS: The paper introduces a unique approach by incorporating collective learning into the MCTS paradigm. This integration allows models to conjecture and identify effective reasoning paths that individual models may struggle with alone.
2. Construction of Mulberry-260k Dataset: Using CoMCTS, the researchers constructed a dataset named Mulberry-260k. This dataset contains a rich set of multimodal inputs, each associated with a well-defined tree of reasoning nodes. It serves as a training ground for MLLMs to develop step-by-step reasoning and reflection capabilities.
3. Development of Mulberry MLLM Series: The Mulberry models are trained using the Mulberry-260k dataset, showcasing enhanced reasoning and reflection abilities. These models outperform many open-source MLLMs and are competitive against some closed-source ones, as demonstrated through extensive experiments on various benchmarks.

Numerical Results and Claims

The empirical evaluation indicates that CoMCTS outperforms traditional MCTS in terms of both search success rate and computational efficiency. The proposed method showed superiority in constructing effective reasoning paths, especially in scenarios where complex multimodal inputs necessitate intricate reasoning processes. The paper quantifies improvements across multiple benchmarks, although specific numerical details and comparative performance metrics against baselines are not exhaustively detailed in this summary.

Implications and Future Directions

The implications of this research are twofold: practical and theoretical. Practically, CoMCTS paves the way for developing more intuitive and robust MLLMs that can handle complex reasoning tasks by learning incrementally and reflectively. Theoretically, the introduction of collective learning into the tree search paradigm demands a reevaluation of traditional self-bootstrapping methods and sets a precedent for further research into collective model dynamics.

For future developments, the integration of reflective reasoning appears promising. The authors propose expanding the tree search framework to include transitions from negative to positive reasoning nodes, fostering a more dynamic reflection mechanism. This could lead to models that not only reason step-by-step but also navigate themselves out of erroneous paths through reflective thinking, marking a step toward more autonomous and intelligent MLLMs.

In conclusion, CoMCTS marks an innovative stride in refining MLLM reasoning abilities, and the subsequent development of the Mulberry series exemplifies the potential of collective learning. The contributions of this paper are expected to significantly impact future research directions in the field of AI and reasoning in LLMs.