Cross Domain Evaluation of Multimodal Chain-of-Thought Reasoning of different datasets into the Amazon CoT Framework

Abstract: While recent work has extended CoT to multimodal settings, achieving state-of-the-art results on science question answering benchmarks like ScienceQA, the generalizability of these approaches across diverse domains remains underexplored. This work presents a comprehensive analysis of Multimodal Chain-of-Thought (Multimodal-CoT) reasoning, evaluating its effectiveness on the A-OKVQA, OKVQA and ChartQA datasets, which requires broad commonsense and world knowledge beyond scientific reasoning. We implement the two-stage framework proposed by Zhang et al. [3], which separates rationale generation from answer inference and integrates vision features through a gated fusion mechanism with T5-based LLMs. Through systematic ablation studies, we analyze the contributions of vision features, rationale quality, and architectural choices. Our findings reveal that while vision integration significantly reduces hallucination in rationale generation, the effectiveness of CoT reasoning varies substantially across question types, with commonsense reasoning presenting particular challenges. This work provides practical insights for researchers implementing multimodal reasoning systems and identifies key areas for future improvement in cross-domain generalization.

• The paper demonstrates a significant performance drop in MM-CoT on open-domain datasets, revealing challenges in numeric, commonsense, and world-knowledge reasoning.
• The methodology adapts the Amazon CoT pipeline with custom preprocessing, prompt engineering, and gated vision-language fusion to standardize diverse dataset formats.
• Experimental results show accuracies between 14.30% and 32.00% on non-science tasks, underscoring the need for domain-specific MM-CoT enhancements.

Cross-Domain Evaluation of Multimodal Chain-of-Thought Reasoning in the Amazon CoT Framework

Introduction

This work investigates the generalizability of Multimodal Chain-of-Thought (MM-CoT) reasoning frameworks, originally designed for scientific question answering (ScienceQA), to diverse datasets necessitating numeric, commonsense, and world-knowledge reasoning. Specifically, the Amazon CoT architecture, characterized by a two-stage rationale generation and answer inference paradigm leveraging T5-based LLMs and gated fusion of vision features, is systematically evaluated on three external datasets: ChartQA, OK-VQA, and A-OKVQA. The forced domain shift testing is motivated by the paucity of comprehensive empirical validation of MM-CoT frameworks beyond structured scientific benchmarks.

Datasets and Their Cross-Domain Challenges

ChartQA, OK-VQA, and A-OKVQA each stress particular facets of multimodal reasoning:

• ChartQA: Consists of synthetically generated charts (bar, pie, line) requiring structured visual interpretation and numeric reasoning over tabular data. The evaluation probes the MM-CoT model's ability to adapt to geometric, symbolically dense visual information versus natural scenes.

  Figure 1: ChartQA features structured, synthetic chart images designed to elicit numeric and trend-based reasoning.

  

  Figure 2: ChartQA queries require stepwise language-based rationales for numerically grounded inferences.

• OK-VQA: Presents open-ended questions on natural images demanding external world knowledge and commonsense. The task evaluates if rationales generated via MM-CoT can transcend the immediate visual context and capture factual knowledge.

  Figure 3: Example of OK-VQA’s open-ended, knowledge-driven question and rationale prompt format.

• A-OKVQA: Encompasses ambiguous, multi-annotator samples enriched with human-written rationales, requiring both visual grounding and integration of world knowledge. The presence of annotated rationales allows assessment of MM-CoT's rationale fidelity and answer selection in open-domain contexts.

  Figure 4: A-OKVQA images include natural scenes necessitating complex, multi-hop reasoning.

  

  Figure 5: A-OKVQA textual data provide multiple rationales per question, enabling consensus evaluation.

Methodological Adaptations

Transitioning the Amazon CoT pipeline to open-domain, multimodal question answering required several adaptations to both data processing and model architecture:

• Data Preprocessing: Non-ScienceQA datasets (ChartQA, OK-VQA, A-OKVQA) lack standardized JSON format and rationale/lecture fields. Custom harmonization scripts transform these sources into unified representations, inject placeholder fields where needed, and normalize answer formats to enable robust batch processing.
• Prompt Engineering: Multiple-choice logic is suppressed for open-ended datasets. For ChartQA and OK-VQA, prompts elicit unrestricted rationales with numeric or categorical answer extraction. Regular expressions and normalization pipelines ensure reliable retrieval of final predictions.
• Vision-Language Fusion: For each dataset, ViT-based image encoders generate feature vectors that are fused with textual tokens using a gated linear projection and concatenation, maintaining multimodal alignment.
• Evaluation Metrics Expansion: Besides standard Exact Match (EM), numerical tolerance, consensus voting (for A-OKVQA/OK-VQA), and semantic similarity scores (using transformer embeddings) are employed to better reflect answer and rationale quality on diverse data formats.

Experimental Results

Performance degrades substantially when MM-CoT is transferred beyond ScienceQA. Empirical findings include:

Dataset	Accuracy (%)
ChartQA	14.30
A-OKVQA	32.00
OK-VQA	21.31
ScienceQA	90.45

• ChartQA (14.30% EM): The lowest accuracy observed, attributed to the symbolic, geometric nature of charts, the reliance on numeric inference (for which T5 is suboptimal), and the lack of multiple-choice prompts. ViT encoders struggle to adequately process fine-grained axes, fonts, and legends, and rationale quality is limited.
• A-OKVQA (32.00% EM): Highest among the external datasets due to its similarity in rationale format to ScienceQA and presence of annotated rationales. Yet, vision encoder mismatches and diluted rationale grounding persist.
• OK-VQA (21.31% EM): Sits between the two extremes, challenged by the need for external world knowledge integration absent from both image and text, confounded by ambiguous, open-ended question formats.

  Figure 6: Comparative accuracy across ScienceQA, ChartQA, OK-VQA, and A-OKVQA in the Amazon MM-CoT framework.

Discussion and Implications

The dramatic drop in performance outside ScienceQA highlights the poor domain transferability of current MM-CoT architectures. Key deficiencies include:

• Vision Encoder Generalization: ViT, trained primarily on natural scenes, is ill-suited to synthetic chart parsing and fails to capture fine-grained, geometry-dependent features.
• Rationale Quality and Hallucination: Vision integration via gated fusion reduces hallucination in rationale generation, but quality varies markedly with task structure; verbose rationales frequently misalign with correct inferential paths in open-domain tasks.
• Numeric and Commonsense Reasoning: T5-based architectures, even when multimodal, struggle with arithmetic and external knowledge synthesis absent from their training corpus. Lack of explicit retrieval or multihop reasoning exacerbates this failure mode.
• Training Resource Constraints: Heavy computational requirements impede practical deployment and extensive fine-tuning on niche datasets, limiting generalization.

Theoretical and Practical Implications

The study identifies critical obstacles to effective cross-domain generalization in MM-CoT frameworks. The inability to adapt stepwise rationales and vision-language fusion mechanisms to open-ended, knowledge-driven, or numerically grounded domains reduces the framework's utility in broader, real-world reasoning contexts. There is limited evidence of transfer from structured science QA to commonsense, chart-based, or consensus-reasoning settings.

Future research should prioritize:

• Incorporation of domain-adaptive vision encoders (e.g., chart-specific or knowledge-grounded models).
• Integration with retrieval-augmented generation for open-domain reasoning.
• Numeric and symbolic rationale pretraining or bootstrapping strategies for chart and table-based inference.
• Efficient, resource-conserving fine-tuning protocols applicable to low-resource deployment scenarios.

Conclusion

Empirical analysis demonstrates that Multimodal Chain-of-Thought frameworks exhibit constrained cross-domain generalization. The Amazon CoT paradigm, when evaluated on ChartQA, OK-VQA, and A-OKVQA, falls markedly short of its ScienceQA performance, with system limitations traceable to vision encoder misfit, prompt and rationale misengineering, and resource bottlenecks. These findings articulate an urgent need for domain-specialized multimodal architectures and prompt engineering methodologies to bridge the gap between structured scientific reasoning and open-domain intelligence.