ProLaViT: Learning Progressive Latent Visual Thoughts in Structured Latent Space

Abstract: Multimodal LLMs (MLLMs) have achieved remarkable progress but still struggle with complex visual reasoning tasks requiring multi-step perception and logical deduction. While explicit visual generation incurs prohibitive computational costs, existing latent approaches often rely on external experts or lack rigorous cognitive logic. In this paper, we introduce ProLaViT (Progressive Latent Visual Thought), a framework empowering MLLMs to perform structured visual derivation in the continuous latent space. Unlike works dependent on heterogeneous external models, ProLaViT leverages an endogenous self-distillation mechanism, utilizing the model's own visual encoder to supervise latent thoughts. To facilitate this, we construct a scalable programmatic synthesis pipeline enabling the model to internalize algorithmic precision without inference time tools. We design two reasoning paradigms: (1) Coarse-to-Fine Causal Chain for spatial tasks, guiding attention from global context to local targets. (2) Dialectical Reasoning Chain for logical tasks, incorporating counter-factual thinking for verification. Furthermore, we propose a Distance-Weighted Diversity Loss to impose topology-aware constraints, preventing feature degeneration by enforcing semantic distinctiveness. Extensive experiments demonstrate that ProLaViT outperforms baselines on vision-centric benchmarks, achieving superior accuracy and interpretability with high efficiency.

• The paper introduces a progressive latent visual reasoning framework that decomposes complex tasks into multi-step chains using a structured latent space.
• It employs endogenous self-distillation and distance-weighted diversity loss to maintain visual fidelity and counteract semantic collapse.
• Experimental results demonstrate significant improvements on benchmarks like ChartQA and BLINK, reducing computational cost while boosting accuracy.

ProLaViT: Progressive Latent Visual Reasoning in Structured Latent Space

Motivation and Background

Multimodal LLMs (MLLMs) continue to extend their capabilities across both vision and language domains, yet they exhibit critical deficiencies on complex visual reasoning tasks that require structured multi-step perception, spatial localization, and logical deduction. Conventional textual Chain-of-Thought (CoT) reasoning is inadequate due to the inherent modality gap—fine-grained visual details and spatial relationships are not faithfully transcribed when projected into text, leading to hallucinations and brittle outputs. Alternatives such as explicit visual generation methods (e.g., generating intermediate images as reasoning traces) are prohibitively expensive in computational cost; on the other hand, existing latent reasoning approaches, typically single-step, lack the capacity to decompose complex tasks and instead suffer from semantic collapse and representational fragility.

Figure 1: Three paradigms for multimodal reasoning. Only progressive latent chains, as in ProLaViT, maintain precise spatial grounding and avoid both hallucination and context loss.

ProLaViT Framework

ProLaViT introduces a methodologically rigorous latent visual reasoning paradigm. The core insight is the necessity of Progressive Visual Derivation: instead of leaping from input to answer in a single reasoning step or generating costly sets of intermediate images, the model operates within a structured latent space, decomposing visually challenging tasks into multi-step chains of latent thoughts. Each step is a distinct, learnable latent state, aligned to well-defined cognitive operations and refined with two principal mechanisms: Endogenous Self-Distillation and Distance-Weighted Diversity Loss.

Figure 2: The ProLaViT architecture, showing progressive latent token generation and endogenous self-distillation against image-synthesized teacher features.

Reasoning Chains

Spatial Reasoning: ProLaViT formalizes visual reasoning as a Coarse-to-Fine Causal Chain (e.g., Locate → Focus → Isolate), where the model initially establishes global spatial context, then incrementally refines its attention toward targets via bounding boxes, spatial cropping, and eventually object segmentation. Each reasoning step is operationalized as a transformation in latent space, supported by tokenized representations.

Logical Reasoning: For tasks such as jigsaw puzzles, ProLaViT adopts a Dialectical Reasoning Chain (Hypothesize → Critique → Verify), embedding counterfactuals and trial-and-error operations into its progressive latent state sequence.

Figure 3: Top—Coarse-to-fine causal chain for spatial reasoning. Bottom—Dialectical chain for logical puzzles; both form explicit, structured latent token paths.

Training Protocols

Endogenous Self-Distillation

One key problem in sequential latent reasoning is the unavailability of ground-truth for intermediate visual states. ProLaViT resolves this by programmatically synthesizing chains of auxiliary images (across canonical operations such as grid overlays, bounding boxes, segmentation, etc.) for each training sample. These images are analytically precise via code, not noisy or heuristic annotations. Frozen visual encoders derived from the native MLLM backbone extract features from these auxiliary images, which serve as teacher representations: the model’s learnable latent thought tokens are distilled to match these features. Thus, the visual chain-of-thought is internalized in latent space without recourse to external models, eliminating domain shift.

Distance-Weighted Diversity Loss

A common issue, latent collapse, manifests as non-distinct representations between steps, rendering the chain redundant. ProLaViT imposes a Distance-Weighted Diversity Loss, penalizing cosine similarity between anchor token centroids of different reasoning steps, with margin-based weighting scaled by causal chain distance. This ensures both local consistency between adjacent steps and global semantic separation for distant reasoning states.

Figure 4: Cosine similarity heatmaps. Only the distance-weighted loss induces hierarchically structured latent chains with contiguous but distinct state transitions.

Curriculum and Optimization

The model is trained in multiple curriculum stages, initializing anchor tokens, aligning visual features, enforcing diversity, and finally integrating general VQA capabilities via joint optimization. LoRA-based parameter-efficient tuning and heavy reliance on the frozen base encoder ensure robust transfer and minimal catastrophic forgetting.

Experimental Evaluation

ProLaViT is benchmarked across MMVP, VisPuzzle, VStar, ChartQA, BLINK, and CV-Bench, directly contrasting against text-based CoT, one-step latent, and tool-based approaches. Key findings:

• Structured decomposition is critical: Progressive latent chains yield significant gains over single-step latent approaches, especially for complex or compositional tasks (e.g., +15.97% on ChartQA, +10% on BLINK-Jigsaw).
• Bridging the modality gap: Latent-space reasoning outperforms text-based CoT on spatially intensive benchmarks by maintaining visual fidelity lost in text abstraction.
• Robust self-distillation: Endogenous alignment using native visual features outperforms external experts (e.g., DINOv2, SDXL-VAE), which suffer from misalignment and poor transfer on non-natural images or structured data.
• Diversity constraint efficacy: Distance-weighted loss preserves chain hierarchy and semantic progression, outstripping uniform margin-based methods, which impair contextual coherence.
• Latency and efficiency: Latent chains reduce inference and training cost by avoiding repeated image encodings and external tool calls required by explicit visual reasoning methods.

Practical and Theoretical Implications

ProLaViT concretely demonstrates that structured, progressive latent reasoning chains not only improve MLLM performance on spatial/logical tasks, but also enable more interpretable intermediate states, amenable to inspection and probing. The endogenous alignment strategy sidesteps the bottlenecks and domain gaps of external experts and supervisees. Notably, moving visual reasoning away from pixel-level or text-level and into the structured latent manifold of the MLLM backbone has direct implications for both efficiency and the unification of cognition over modalities.

The paradigm established by ProLaViT paves the way for future research in temporally extended tasks (e.g., video reasoning), complex geometric manipulations, compositional task generalization, and hierarchical cognitive modeling within foundation models. This approach is also suited for integration with dynamic environments (e.g., robotics), where computational efficiency and real-time reasoning are paramount.

Conclusion

ProLaViT fundamentally advances multimodal LLMs by instilling progressive latent visual reasoning chains, enabling fine-grained, stepwise visual cognition in continuous space at high efficiency and fidelity. Its endogenous self-distillation obviates the need for external vision pipelines, while topologically-aware diversity regularization enforces robust, non-collapsing latent trajectories. Empirical validation across challenging perception and logic benchmarks establishes new state-of-the-art accuracy with superior computational efficiency, and points toward a rich field of future research across both visual and generalized multimodal reasoning.