Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSampling

Abstract: In the unsupervised self-evolution of Multimodal LLMs, the quality of feedback signals during post-training is pivotal for stable and effective learning. However, existing self-evolution methods predominantly rely on majority voting to select the most frequent output as the pseudo-golden answer, which may stem from the model's intrinsic biases rather than guaranteeing the objective correctness of the reasoning paths. To counteract the degradation, we propose Continuous Softened Retracing reSampling (CSRS) in MLLM self-evolution. Specifically, we introduce a Retracing Re-inference Mechanism (RRM) that the model re-inferences from anchor points to expand the exploration of long-tail reasoning paths. Simultaneously, we propose Softened Frequency Reward (SFR), which replaces binary rewards with continuous signals, calibrating reward based on the answers' frequency across sampled reasoning sets. Furthermore, incorporated with Visual Semantic Perturbation (VSP), CSRS ensures the model prioritizes mathematical logic over visual superficiality. Experimental results demonstrate that CSRS significantly enhances the reasoning performance of Qwen2.5-VL-7B on benchmarks such as MathVision. We achieve state-of-the-art (SOTA) results in unsupervised self-evolution on geometric tasks. Our code is avaible at https://github.com/yyy195/CSRS.

• The paper introduces CSRS, which integrates retracing re-inference, continuous softened rewards, and visual perturbation to counteract bias-induced model collapse.
• It reports significant performance gains on geometric and mathematical benchmarks, including a peak accuracy of 68.25% and improvements of over 2% on key datasets.
• Ablative analysis confirms that each component of CSRS is critical for preserving logical diversity and preventing deterministic overfitting in self-evolutionary training.

Stabilizing Unsupervised Self-Evolution of Multimodal LLMs via Continuous Softened Retracing reSampling

Motivation and Theoretical Framework

Unsupervised self-evolutionary reinforcement learning in Multimodal LLMs (MLLMs) is a compelling strategy for continual model improvement without reliance on expensive annotated data. Traditional methods often employ majority voting to select pseudo-labels; these signals are derived from model-internal frequency estimates and do not guarantee logical correctness. This self-reinforcement of initial bias via binary feedback fosters deterministic mapping, narrows the solution space, and accelerates model collapse.

Figure 1: CSRS contrasts with majority voting: the latter amplifies model biases, whereas CSRS explicitly counters collapse via retracing and reward softening.

The current study introduces Continuous Softened Retracing reSampling (CSRS) to address this problem. CSRS augments the self-evolution loop by incorporating a retracing re-inference procedure, softened frequency rewards, and visual semantic perturbations. The theoretical underpinning is grounded in the Group Relative Policy Optimization (GRPO) framework, with CSRS analytically shown to mitigate exponential polarization induced by majority voting. The derivation demonstrates that, compared to 0-1 binary rewards, continuous rewards injected by CSRS reduce distributional entropy decay and maintain exploration of long-tail reasoning paths.

CSRS: Architecture and Key Mechanisms

The CSRS pipeline builds on several synergistic components:

• Retracing Re-inference Mechanism (RRM): Maternal (initial) reasoning trajectories are explored; anchor points are identified in each trajectory. At these retracing anchor locations, new local inferences (re-inference) are sampled. This process explicitly increases coverage of critical long-range logical states and ensures local exploration at pivotal steps where early trajectory bias might otherwise propagate uncorrected.
• Softened Frequency Reward (SFR): Instead of a binary pseudo-gold signal, SFR calculates rewards based on statistical answer frequencies in the union of initial and re-inference distributions, modulated by changes in answer support induced by resampling. This formulation dampens mode amplification, channels reward sensitivity toward robust but rare reasoning paths, and prevents exclusive optimization for the most frequent, not necessarily correct, answers.
• Visual Semantic Perturbation (VSP): Gaussian noise is injected into images during re-inference to decouple superficial pattern matching from deeper logical reasoning. The model is forced to utilize underlying mathematical structures rather than low-level visual cues.

  Figure 2: CSRS pipeline: maternal trajectory generation, anchor retracing, local resampling, reward calibration, and visual perturbation are iteratively orchestrated.

Empirical Results

Extensive experiments are conducted using Qwen2.5-VL-7B as the base architecture, with self-evolution post-training on major geometric reasoning datasets (Geometry3K, GeoQA, MMR1) and evaluation on MathVision, MathVerse, MathVista, and We-Math. CSRS yields consistent improvements over both the vanilla backbone and prevailing unsupervised SOTA (e.g., MM-UPT), with precise gains such as +2.34% on MathVision, +2.41% on MathVista, and a peak accuracy of 68.25% in the unsupervised regime.

Ablative analysis confirms the necessity of each CSRS component. Performance declines sharply if SFR, RRM, or VSP are individually omitted, evidencing non-trivial synergistic effects. Additionally, robustness is maintained across multiple MLLM families and varying retracing/sampling configurations.

Accuracy analysis (Figure 3) shows the proportion of correct answers increases for both high- and low-frequency intervals throughout training, with model collapse—as measured by overconcentration on a narrow set of high-confidence samples—substantially alleviated relative to majority voting.

Figure 3: Model accuracy is elevated across all answer frequency bands when using CSRS-guided self-evolution.

Visualization of maternal and re-inference answer distributions reveals that maternal sets quickly contract into a narrow semantic region under majority voting, whereas CSRS maintains broader distributional support, actively populating long-tail solution clusters throughout training steps (Figure 4).

Figure 4: Distributional drift: blue (maternal) clusters collapse, while orange (CSRS-augmented) maintain semantic diversity over training epochs.

Entropy measurements further corroborate the stabilization effect, as CSRS achieves slower decay, sustaining higher-entropy policy distributions and thus greater logical diversity (Figure 5).

Figure 5: Entropy change: CSRS preserves higher entropy, moderating the determinism surge observed in conventional self-training.

Case Analysis and Error Correction

Case studies in geometric reasoning emphasize the practical efficacy of the retracing module in recovering logically valid but underrepresented solution paths. For example, where the majority logic converges erroneously, re-inference from anchor points—especially with SFR-reweighted signals—amplifies rare but correct answers. In contrast, persistent logical bias in baseline methods reinforces consistent, yet wrong, trajectories, obstructing correction (Figure 6).

Figure 6: Case studies demonstrate that retracing-plus-softened-reward enables the surfacing of valid rare solutions that majority voting suppresses.

Implications and Prospects

CSRS addresses a fundamental obstacle to reliable unsupervised self-evolution in MLLMs: pathological feedback collapse and trapped exploration. The introduction of local retracing and continuous rewards as stabilizers has several implications:

1. Generalizability: Improvements are consistent across backbone architectures and datasets, suggesting the approach will scale to other MLLM variants and complex multimodal domains.
2. Exploratory resilience: By preserving entropy and emphasizing long-tail reward, CSRS pushes against mode collapse, a challenge especially acute in high-complexity reasoning settings.
3. Automatic curriculum creation: CSRS moves toward process-level supervision, dynamically identifying decision pivots and amplifying their corrective signals, indicating possible future extension to curriculum-aware or agentic self-improvement protocols.
4. Limitation and future work: CSRS is currently evaluated on mathematical reasoning; extending similar stabilization to broader open-ended multimodal understanding still requires further investigation. Advancing toward intrinsic reward mechanisms that incorporate dynamic, self-reflective evaluation is a direction for next-generation agent self-evolution.

Conclusion

This work proposes and empirically validates Continuous Softened Retracing reSampling as an effective method for stabilizing unsupervised self-evolution in MLLMs. By integrating retracing anchor-based resampling, frequency-sensitive reward modulation, and visual perturbation, CSRS eliminates severe collapse behaviors, enhances logical diversity, and achieves SOTA performance on multimodal math benchmarks. The approach provides a mathematically rigorous and practically robust avenue for sustaining open-ended model improvement in the absence of manual labels, establishing new paradigms for autonomous agent development in complex cognitive domains.

Reference: (2604.03647)