Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

Abstract: Multimodal LLMs (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the modality gap is task- and data-dependent. For example, math tasks degrade by over 60 points on synthetic renderings, while natural document images often match or exceed text-mode performance. Rendering choices such as font and resolution are strong confounds, with font alone swinging accuracy by up to 47 percentage points. To understand this, we conduct a grounded-theory error analysis of over 4,000 examples, revealing that image mode selectively amplifies reading errors (calculation and formatting failures) while leaving knowledge and reasoning errors largely unchanged, and that some models exhibit a chain-of-thought reasoning collapse under visual input. Motivated by these findings, we propose a self-distillation method that trains the model on its own pure text reasoning traces paired with image inputs, raising image-mode accuracy on GSM8K from 30.71% to 92.72% and transferring to unseen benchmarks without catastrophic forgetting. Overall, our study provides a systematic understanding of the modality gap and suggests a practical path toward improving visual text understanding in multimodal LLMs.

• The paper demonstrates that rendering artifacts (e.g., font and resolution changes) drastically reduce image-mode performance in tasks like GSM8K.
• It highlights error types dominated by reading failures while preserving higher-order reasoning, as evidenced by a 1.5x increase in calculation errors.
• The study introduces self-distillation using text-mode chain-of-thought traces, which nearly eliminates the performance gap without catastrophic forgetting.

Diagnosing and Bridging the Modality Gap in Multimodal LLMs

Overview

This paper rigorously investigates the "modality gap" in multimodal LLMs (MLLMs) when textual information is transformed into pixels and fed as visual input, rather than as textual tokens. Through extensive empirical evaluation across seven prominent benchmarks, five distinct input modes, and seven state-of-the-art MLLMs, the authors systematically uncover the roots of performance degradation in image-mode text reasoning. The study highlights the differentiated robustness of models, the confounding impact of rendering choices (font, resolution, color), and provides a grounded taxonomy of error types. Furthermore, they introduce self-distillation as an effective remediation strategy, closing the modality gap in demanding benchmarks such as GSM8K, and demonstrating transferability without catastrophic forgetting.

Figure 1: Humans read text through vision from diverse sources such as books, webpages, and documents, whereas MLLMs may yield different predictions when the same textual content is presented in different input forms.

Modalities, Benchmarks, and Models

The paper evaluates seven MLLMs, including both proprietary (e.g., GPT-5.2) and open-source (e.g., Qwen2.5-VL, InternVL3, Pixtral-12B) models. The evaluation suite encompasses both synthetic and naturally occurring visual text, spanning benchmarks such as MMLU, ARC, GPQA, GSM8K, HumanEval, SQuAD v2, and QASPER. Five input modes are defined:

• Pure Text (canonical token input)
• Pure Image (synthetically rendered text)
• Instr.+Image (Visual QA-style; image + textual instruction)
• OCR-1P and OCR-2P (diagnostic settings isolating reading and reasoning)

  Figure 2: Diagnosing the modality gap in visual text understanding. The error taxonomy shows image modality amplifies reading/calculation errors and suppresses chain-of-thought but leaves reasoning intact. Remedies include rendering controls and LM-only self-distillation.

Modality Gap: Task and Distributional Effects

Performance gaps between text and image input are shown to be highly task- and distribution-dependent. On synthetic renderings of challenging math and coding tasks, image-mode accuracy drops by over 60 points. For instance, on GSM8K, Qwen3-VL-8B drops from 93.56% (text) to 30.71% (image). By contrast, for realistic document images (e.g., arXiv PDFs, Wikipedia screenshots), models often achieve parity or exceed text-mode performance, indicating distributional alignment with pretraining corpora.

Rendering choices are revealed as strong confounds: a change in font can swing accuracy by up to 47%, handwriting-style renderings are particularly detrimental, and inappropriate image resolution leads to sharp performance drops except in models equipped with specialized resolution handling (e.g., InternVL3.5 with Visual Resolution Router). Compact renderings yield higher performance, suggesting that preserving critical visual features is more important than pixel volume.

Error Taxonomy: Reading vs. Reasoning

A grounded-theory analysis classifies over 4,000 model errors and reveals the error profile is dominated by reading failures in image mode:

• Calculation/Mathematical and Format/Output errors increase by 1.5x and ~1.35x under image input, respectively.
• Conceptual/Factual Recall and Reasoning errors are largely invariant across modalities.
• Chain-of-thought reasoning collapse is observed: models output much shorter responses in image mode, with step-by-step reasoning nearly suppressed in GSM8K.

Figure 3: Error distribution by input mode indicates image modality selectively amplifies perceptual errors (reading/calculation/formatting) but leaves knowledge and reasoning unaffected.

Diagnostic Isolation and OCR Limitations

OCR-1P (prompted extraction followed by reasoning in one pass) catastrophically fails in many models, confirming that single-pass reading-reasoning pipelines are unnatural. OCR-2P (two-pass extraction and reasoning) recovers substantial performance but destroys code-generation quality; structure and formatting critical for code are lost. Global OCR metrics (word/character error rate) correlate weakly ($r = 0.238$) with downstream accuracy. Fine-grained errors on task-critical tokens seem more impactful.

Figure 4: Task accuracy vs.\ CER shows a weak correlation, underscoring that aggregate OCR errors do not explain task performance.

Bridging the Gap: Self-Distillation

To remediate both the perceptual deficit and reasoning collapse, the authors propose self-distillation: training models with their own text-mode chain-of-thought traces as supervision for image-mode inputs. GSM8K image-mode accuracy surges from 30.71% to 92.72% with LM-only LoRA fine-tuning (filtered traces), nearly matching text-mode accuracy and reducing the gap from 62.85 to 1.37 percentage points.

• Filtering CoT traces for correctness is unnecessary; imperfect traces suffice.
• Vision encoder adaptation (ViT-only) provides partial improvement (up to 85.29%), but LM adaptation is crucial for full recovery.
• Catastrophic forgetting is absent: models retain or improve performance on other benchmarks after distillation.

Implications and Future Directions

The findings fundamentally reposition the modality gap as a training-distribution phenomenon, rather than an inherent limitation of multimodal reasoning. Perceptual and rendering artifacts are shown to be the primary impediments—not failures in higher-order cognition or logical inference. The suppression of chain-of-thought is tractable by distillation, and cross-modal transfer is robust.

The practical implications are significant:

• Benchmarking MLLMs must rigorously account for visual rendering confounds.
• Model training protocols should include targeted visual-text distillation to synchronize perceptual and cognitive pathways.
• Further architectural refinements (e.g., vision-centric tokenization, unified multimodal transformers) can augment robustness, but minimal LM fine-tuning suffices for substantial gains.
• Richer calibration on realistic document imagery should be prioritized over synthetic renderings.

On the theoretical side, the paper supports a modular interpretation of MLLM failure modes, where visual impairment is separable from reasoning impairment. This insight enables new strategies in cross-modal knowledge transfer and distillation.

Conclusion

This work systematically diagnoses the factors underlying the modality gap in MLLMs, demonstrating that reading failures—due to rendering mismatches—are the central cause of degraded performance on visual text. Cognitive and reasoning faculties remain intact but are suppressed by visual input. By introducing self-distillation of text-mode reasoning traces, the authors show performance gaps can be closed efficiently, obviating the need for architectural overhaul. The results prompt the community to reassess visual text benchmarks, focus training on practical distribution alignments, and explore further cross-modal distillation, with broader ramifications for robust, scalable multimodal language understanding.