Multimodal Pathology LLM

• The paper introduces MPLLM, showing that merging visual intermediate observations with chain-of-thought reasoning enhances accuracy in spatial and dynamic tasks.
• It leverages dual subroutines—world reconstruction and simulation—to generate detailed visual representations that reduce ambiguity compared to text-only models.
• Empirical evaluations demonstrate substantial efficiency gains, with visual chain-of-thought achieving higher accuracy using fewer samples than traditional verbal models.

A Multimodal Pathology LLM (MPLLM) is a unified multimodal model (UMM) integrating both visual and verbal generative capabilities in service of human-like reasoning, particularly for tasks where traditional verbal-only LLMs encounter representational bottlenecks. These models leverage visual generation as an internal world model for chain-of-thought (CoT) reasoning, enabling more effective solutions to problems grounded in the physical and spatial domains—such as the reconstruction of 3D structures, simulation of physical dynamics, and spatial state-tracking—where mental imagery outperforms symbolic manipulation. The principle undergirding such models is the Visual Superiority Hypothesis: for multimodal reasoning grounded in the physical world, images generated as intermediate world-model observations confer richer, less ambiguous state representations than those producible by pure text or symbolic verbal traces (Wu et al., 27 Jan 2026).

1. Foundations: Visual Superiority Hypothesis and Theoretical Framework

The Visual Superiority Hypothesis posits that visual generation, serving as an internal world model, unlocks more informative and knowledge-rich representations compared to purely verbal world models in multimodal reasoning tasks centered on physical reality. In contexts requiring mental reconstruction (e.g., 3D rotations, unfolding objects, prediction of physical dynamics), explicit image generation acts as a “visual scratchpad,” reducing uncertainty around latent states and facilitating subsequent reasoning steps (Wu et al., 27 Jan 2026). This hypothesis finds partial support in multimedia learning theories, which argue for dual-channel processing with synergistic effects when words and pictures are integrated, but caution against the assumption of universal superiority—active integration of pictorial and textual information is critical, and picture bias may arise when attention is drawn away from core verbal components (Ogren et al., 2017).

Within MPLLM architectures, reasoning is formalized in terms of multi-observable Markov decision processes (MOMDPs):

$$\mathcal{M}=(\mathcal{S},\mathcal{A},p,\Phi,\mathcal{O}_\phi,e_\phi)$$

where $\mathcal{S}$ is the latent state space, $\mathcal{A}$ the action space, $p$ the transition kernel, $\Phi$ observation parameters, and $e_\phi(s)$ the view (visual or verbal) of state $s$. Reasoning traces consist of paired textual steps and model-generated intermediate observations, which may be explicit (visual or verbal) or implicit (no explicit state) (Wu et al., 27 Jan 2026).

2. Atomic World-Modeling Capabilities

MPLLMs decompose their multimodal reasoning capabilities into two atomic subroutines: world reconstruction and world simulation.

• World Reconstruction involves novel-view synthesis, denoted

$$p_\theta(o_{\phi_{n+1}}|o_{\phi_1},\dots,o_{\phi_n}),$$

where the model infers a new visual observation from prior views.

• World Simulation supports future-step prediction,

$p_\theta(o_{t+1}|o_{\le t},a_{\le t}),$

forecasting subsequent observations given previous states and actions.

This functional partitioning allows for explicit chaining of intermediate images (visual WM) within CoT reasoning, supplementing or superseding symbolic and textual representations (verbal WM) (Wu et al., 27 Jan 2026).

3. Information-Theoretic and Learning-Theoretic Arguments

Rigorous analysis reveals two central theoretical advantages for visual world models:

• Informativeness Trade-off: The error in answering a reasoning problem is upper-bounded by the sum of reasoning and world-modeling errors:

$$\mathrm{KL}(p(A|Q,I)\|p_\theta(A|Q,I)) \leq \sum_{i=1}^{H+1}\mathbb{E}_p[\mathrm{KL}(p(r_i|R_i)\|p_\theta(r_i|R_i))] + \sum_{i=1}^H\mathbb{E}_p[\mathrm{KL}(p(o_i|\tilde R_i)\|p_\theta(o_i|\tilde R_i))]$$

Explicitly generating observations $o_i$ can reduce uncertainty about what reasoning step $r_i$ to take. The benefit is bounded by the mutual information that observations encode about the latent state and what the reasoning demands:

$$\mathbb{I}(o_{1:i-1}; r_i|\cdot ) \leq \min(\mathbb{I}(o_{1:i-1}; s_{1:i-1}), \mathbb{I}(r_i; s_{0:i-1}, r_{0:i-1}))$$

Visual observations generally have greater $\mathbb{I}(o; s)$ (geometric, structural, and dynamic content) with less need for inter-channel translation than text (Wu et al., 27 Jan 2026).

• Prior-Knowledge Transfer: Generalization error is minimized when the modality-specific distribution shift is small. Visual world models, trained on large-scale Internet data, inherit strong priors on physical dynamics and spatial transformations, accelerating adaptation to downstream tasks where explicit knowledge is scarce and verbal priors are weak (Wu et al., 27 Jan 2026).

4. Benchmarks and Task Taxonomy: VisWorld-Eval

The VisWorld-Eval evaluation suite operationalizes the distinction between world simulation (physical dynamics) and world reconstruction (structure):

Category	Task Example	Visual Model Advantage
World Simulation	Paper Folding, Ball Tracking, Multi-Hop Manipulation	High
World Reconstruction	Cube 3-View Projection, Real-World Spatial Reasoning	High
Controls (low-dim)	Maze, Sokoban	None

Tasks favor visual world modeling when symbolic descriptions are ambiguous or require tedious encoding, and when verbal priors fail to encode dynamics or 3D geometry (Wu et al., 27 Jan 2026).

5. Empirical Evaluations: Superiority, Sample Efficiency, and Fidelity

Supervised experiments on leading UMMs demonstrate substantial gains for visual CoT over verbal CoT:

• Accuracy Gains (World Simulation): Visual CoT produces 50–80 percentage-point improvements (e.g., paper folding: verbal ≈ 30%, visual ≈ 80%) (Wu et al., 27 Jan 2026).
• Reconstruction Tasks: On cube 3-view, visual CoT achieves ~20-point superiority across stack sizes; for MMSI, improvements are 10–15 points (Wu et al., 27 Jan 2026).
• Control Tasks: Maze and Sokoban show no significant advantage for visual CoT, with implicit representations achieving 60–70% accuracy (Wu et al., 27 Jan 2026).
• Sample Efficiency: Visual CoT attains 60% accuracy with 500 samples, whereas verbal CoT requires >2000 samples (a 4× efficiency gain) (Wu et al., 27 Jan 2026).
• World-Model Fidelity: Verbal CoT yields near-zero structural fidelity in generated views; visual CoT exceeds 50% shape fidelity even for unseen configurations (Wu et al., 27 Jan 2026).
• Emergent Implicit WM: In maze tasks, UMMs encode latent coordinates at >95% accuracy, explaining the lack of visual CoT benefit for low-dimensional domains (Wu et al., 27 Jan 2026).
• Reinforcement Learning: RL from verifiable rewards improves performance in all CoT styles but does not close the visual-verbal gap; visual advantages persist at ~20 points even after RL (Wu et al., 27 Jan 2026).

6. Human–AI Comparisons and Serial Processing Bottlenecks

Distinct from multimedia education effects, studies on Vision LLMs (VLMs) demonstrate systematic deficiencies in visually-grounded serial processing (Budny et al., 29 Sep 2025):

• Serial Load Impact: When serial attentional operations (object individuation, incremental mental transformations) are needed, human accuracy remains near ceiling but reaction time increases. VLMs, lacking these routines, exhibit sharp declines in accuracy as task complexity rises.
• Quantitative Predictors: There is a strong negative correlation between human RT (an index of serial processing demand) and VLM accuracy (geometric reasoning: $r=-0.73$, mental rotation: $r=-0.88$) (Budny et al., 29 Sep 2025).
• Implications: Augmenting VLMs with linguistic CoT or image tool use only partly mitigates these deficits for certain referential tasks. The absence of region-grounded, sequential attention and transformation in current architectures remains a limiting factor (Budny et al., 29 Sep 2025).

7. Design Principles and Future Directions

Technical insights from empirical and theoretical analyses yield several design implications for next-generation MPLLMs:

• Integration of Interleaved Visual-Verbal CoT: Unified architectures should natively support chained visual and verbal reasoning steps within their backbone UMMs (Wu et al., 27 Jan 2026).
• Investment in Visual Generation Quality: Enhancement of diffusion or flow-based visual generation to improve fidelity of intermediate world-model observations (Wu et al., 27 Jan 2026).
• RL Algorithms for Joint Reasoning: Development of RL methods that jointly optimize across verbal and visual tokens, not exclusively text (Wu et al., 27 Jan 2026).
• Task Curriculum: Explicitly designed training curricula requiring both world reconstruction and simulation for robust multimodal world-modeling capacity (Wu et al., 27 Jan 2026).
• Serial Reasoning Modules: Augmentation with reinforcement-learned fixation, pointer policies, or attention mechanisms that approximate human saccade-and-fixate cycles to bridge serial processing gaps (Budny et al., 29 Sep 2025).

A plausible implication is that future embodied agents and medically oriented pathology models should employ interleaved multimodal reasoning, especially for tasks involving spatial and physical inference, in order to close the gap with human performance. However, the naïve expectation of universal multimedia advantage is refuted—active, targeted integration of visual and textual modalities is essential for measurable benefit (Ogren et al., 2017).