SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMs

Abstract: Multimodal LLMs (MLLMs) have made remarkable progress in multimodal perception and reasoning by bridging vision and language. However, most existing MLLMs perform reasoning primarily with textual CoT, which limits their effectiveness on vision-intensive tasks. Recent approaches inject a fixed number of continuous hidden states as "visual thoughts" into the reasoning process and improve visual performance, but often at the cost of degraded text-based logical reasoning. We argue that the core limitation lies in a rigid, pre-defined reasoning pattern that cannot adaptively choose the most suitable thinking modality for different user queries. We introduce SwimBird, a reasoning-switchable MLLM that dynamically switches among three reasoning modes conditioned on the input: (1) text-only reasoning, (2) vision-only reasoning (continuous hidden states as visual thoughts), and (3) interleaved vision-text reasoning. To enable this capability, we adopt a hybrid autoregressive formulation that unifies next-token prediction for textual thoughts with next-embedding prediction for visual thoughts, and design a systematic reasoning-mode curation strategy to construct SwimBird-SFT-92K, a diverse supervised fine-tuning dataset covering all three reasoning patterns. By enabling flexible, query-adaptive mode selection, SwimBird preserves strong textual logic while substantially improving performance on vision-dense tasks. Experiments across diverse benchmarks covering textual reasoning and challenging visual understanding demonstrate that SwimBird achieves state-of-the-art results and robust gains over prior fixed-pattern multimodal reasoning methods.

• The paper introduces a hybrid autoregressive MLLM that dynamically switches between text-only, vision-only, and interleaved reasoning modes.
• It employs a dynamic, resolution-aware visual token budget and a curated multi-mode fine-tuning dataset to optimize performance.
• Empirical results on numerous benchmarks demonstrate state-of-the-art performance in both perception-intensive and symbolic reasoning tasks.

SwimBird: Query-Adaptive Switchable Reasoning in Hybrid Autoregressive MLLMs

Introduction

SwimBird introduces a paradigm shift in Multimodal LLMs (MLLMs) by enabling instance-adaptive switching among reasoning modalities—text-only, vision-only, and interleaved modalities—based on input requirements. Prior MLLMs typically rely on pre-determined reasoning patterns—textual Chain-of-Thought (CoT), latent visual reasoning, or fixed alternations—which often induce a mismatch between the query's modality demands and the model's internal computation, leading to degraded performance on both text-centric and vision-intensive tasks. SwimBird addresses these deficiencies by adopting a hybrid autoregressive generation mechanism and a data curation strategy that fosters flexible, efficient, and contextually appropriate reasoning.

Figure 1: SwimBird enables query-adaptive multimodal reasoning by dynamically switching among text-only, vision-only, and interleaved vision–text modes, allowing it to match the modality of reasoning with the query's needs and avoid redundancy or mismatch.

Methodology

Hybrid Autoregressive Architecture

SwimBird unifies two generation protocols:

• Textual reasoning steps use canonical next-token autoregressive prediction, optimized with cross-entropy loss.
• Visual reasoning steps generate continuous latent visual embeddings via next-embedding prediction, supervised with MSE loss towards semantically grounded visual representations.

Both reasoning modes are interleaved in a single autoregressive sequence, mediated by special markers such as <|latent_start|> and <|latent_end|>, enabling the model to conditionally emit either textual tokens or visual embeddings. The mode decisions are entirely data-driven and dynamically resolved at inference, integrating the reasoning mode selection into the model’s generative process.

Figure 2: SwimBird's hybrid autoregressive formulation switches between next-token prediction (for text) and next-embedding prediction (for visual thoughts), dynamically selecting reasoning modes during inference.

Adaptive Visual-Thought Budget

Unlike previous methods that enforce a fixed-length latent token sequence for visual reasoning steps, SwimBird employs a dynamic, resolution-aware token budget. The model determines at runtime both whether to use visual thoughts and the appropriate number of visual tokens, reflecting the perceptual complexity and granularity of the input. This avoids both underfitting (due to insufficient visual capacity) and overfitting/redundancy (wasting resources on unneeded visual computation), and is further reinforced via independent latent token limits for input and intermediate images according to their resolutions.

Switchable Reasoning Data Curation: SwimBird-SFT-92K

A novel supervised fine-tuning set (SwimBird-SFT-92K) is constructed by curating and reason-mode labeling multimodal CoT data from multiple sources. Each instance is categorized into three modes: text-only, vision-only, or interleaved, by evaluating the benefit of intermediate visual hints (using a pass@8 metric) and filtering for cases where each reasoning pattern is provably necessary for the correct solution. This multi-mode supervision is critical for learning effective mode selection and avoiding data-induced bias favoring any one pattern.

Experimental Results

SwimBird is evaluated on a comprehensive suite of vision-centric, multimodal reasoning, and general VQA benchmarks. Comparisons are made against leading text-based MLLMs, latent visual reasoning models, and agentic multimodal systems.

• Fine-grained and high-resolution visual tasks: SwimBird achieves state-of-the-art results on V* Bench (85.5), HR-Bench 4K (79.0), and HR-Bench 8K (74.9), outperforming both text-centric LLMs (e.g., Qwen3-VL-8B-Instruct at 83.8/76.5/71.3) and prior latent visual/agentic models (e.g., DeepEyesV2 at 81.8/77.9/73.8). Empirically, SwimBird’s gains are pronounced on benchmarks with dense perceptual and spatial reasoning requirements, attributed to its adaptive vision-only or interleaved inference.
• General multimodal reasoning: On MMStar, RealWorldQA, WeMath, DynaMath, and MathVerse_MINI, SwimBird consistently ranks first or near-first across both text-centric (“symbolic”) and vision-text hybrid tasks, indicating no trade-off between vision-centric gains and symbolic logic preservation.

Ablation studies reveal that moderate max latent token budgets ($N_{\max}=32$) and balanced reconstruction loss weighting ($\lambda_\text{vis}=0.2$) yield the strongest cross-benchmark robustness, while excessive visual token or loss allocation introduces interference in textual reasoning.

Mode Selection Analysis

SwimBird’s adaptive behavior is quantitatively confirmed by analyzing reasoning-mode selections across tasks.

Figure 3: Distribution of reasoning mode usage by SwimBird, showing alignment of selected mode with benchmark perceptual or symbolic complexity.

• Text-dominant tasks (e.g., DynaMath, MathVerse_MINI) trigger the text-only mode almost exclusively, minimizing unnecessary visual computation and protecting logical chain fidelity.
• Vision-dense perception tasks (e.g., HR-Bench 8K) elicit vision-only and interleaved modes in proportion to perceptual load.
• Mixed-reasoning benchmarks (e.g., WeMath) present a balanced usage of all three modes.

Qualitative studies demonstrate SwimBird’s contextual flexibility:

Figure 4: Case studies showing query-adaptive use of vision-only, text-only, and tightly interleaved modes in SwimBird for distinct problem types.

• For spatial or perceptual reasoning, the model adopts vision-only mode with dynamic latent allocation.
• For pure symbolic/logical applications, it remains in text-only mode.
• For queries requiring both, it interleaves modalities, switching as needed.

Implications and Future Directions

SwimBird’s switchable reasoning architecture offers a powerful and general template for future MLLMs. The decoupling of reasoning modality from fixed scheduling not only improves accuracy and efficiency across heterogeneous tasks but also mitigates error propagation caused by modality mismatch. SwimmingBird’s autonomy in allocating computational budget opens research avenues in neural routing, instance-level reasoning diagnostics, and more nuanced self-adaptive attention mechanisms in multimodal transformers.

From a practical standpoint, such models can underpin more robust, resource-efficient, and interpretable systems in domains ranging from medical imaging triage and scientific diagram analysis to real-world situational understanding, where perceptual and symbolic logic loads are highly variable across instances.

Conclusion

SwimBird establishes a new standard for multimodal reasoning by integrating query-adaptive switching among text-only, vision-only, and interleaved modes, underpinned by a hybrid autoregressive modeling mechanism and a curated multi-mode fine-tuning dataset. Empirical results demonstrate robust, state-of-the-art performance across both perception-intensive and reasoning-driven benchmarks. The switchable reasoning mode mitigates the modality mismatch in previous MLLMs and paves the way for next-generation multimodal systems with dynamic, input-aware cognitive strategies.