CAST: Collapse-Aware multi-Scale Topology Fusion for Multimodal Coreset Selection

Abstract: The training of large multimodal models fundamentally relies on massive image-text datasets, which inevitably incur prohibitive computational overhead. Dataset selection offers a promising paradigm by identifying a highly informative coreset. However, existing approaches suffer from two critical limitations: (i) single-modality-dominated sampling methods, which ignore the fine-grained cross-modal information imbalance inherent in multimodal datasets and thus lead to semantic loss in the other modality; and (ii) coarse-grained sample-scoring-based sampling methods, where the selected coreset tends to be biased toward the scoring model, making it difficult to guarantee distributional equivalence between the coreset and the original dataset. Meanwhile, existing distribution matching and discrete sampling strategies often fail to jointly account for global semantic structure, local fine-grained details, and redundancy-aware coverage in dense regions. To this end, we propose CAST, a Collapse-Aware multi-Scale Topology fusion framework for multimodal coreset selection. We first construct image- and text-modality topologies, and derive a unified topology via local-collapse-aware refinement and cross-modal fusion. We then introduce a multi-scale distribution matching criterion in the diffusion wavelet domain, encouraging the coreset to approximate the original dataset at multiple scales. Finally, we introduce a local soft relational coverage mechanism that extends pure geometric coverage to relation-aware indirect coverage, penalizing redundant selections in dense clusters. Extensive experiments on Flickr30K and MS-COCO show that CAST outperforms existing dataset selection baselines, showcasing great superiority in cross-architecture generalization and energy efficiency over state-of-the-art multimodal synthesis methods.

• The paper introduces CAST, a framework that fuses multi-scale topology to mitigate modality collapse and enable efficient coreset selection.
• It employs diffusion wavelet matching and soft relational coverage to preserve both global structure and local semantic details during compression.
• Experimental results show superior image-text retrieval, robust cross-architecture performance, and significant energy savings compared to existing methods.

CAST: Collapse-Aware Multi-Scale Topology Fusion for Multimodal Coreset Selection

Introduction and Motivation

The increasing scale of multimodal LLMs necessitates ever-larger image-text datasets, yet this trend faces significant computational and energy barriers. Conventional dataset compression via coreset selection offers theoretical and practical advantages, but existing methods are limited by two key flaws: (1) modality-dominant approaches that ignore cross-modal information imbalance, causing semantic loss, and (2) coarse-grained sampling that introduces model or heuristic bias, hindering reliable distributional equivalence. Furthermore, prior graph-based or distribution matching techniques typically fail to simultaneously address global structure, local semantic detail, and redundant proxy coverage in dense regions of the data manifold.

The CAST (Collapse-Aware multi-Scale Topology fusion) framework directly addresses these limitations through novel multi-scale, modality-aware, topology-driven criteria for coresets in multimodal data. The innovations in CAST include a topology refinement and fusion mechanism that explicitly counters modality collapse, multi-scale diffusion wavelet matching for topology-preserving coreset optimization, and a soft relational coverage mechanism that penalizes redundant selection, thus maximizing diversity under aggressive compression budgets.

Fine-Grained Cross-Modal Topology and Information Imbalance

CAST begins with a precise characterization of the multimodal data's heterogeneous information density. The framework constructs modality-specific graphs reflecting local similarity based on image-level and text-level features, using $k$-NN graphs with UMAP-based scale estimation for region-specific locality. Two structural degradation metrics—relation redundancy and neighborhood inseparability—quantify local collapse, informing bidirectional cross-modal refinement. This process allows reliable local topologies from one modality to selectively correct degraded neighborhoods in the other, resulting in cross-modal graphs robust to fine-grained information imbalance.

Figure 1: CAST exposes critical information imbalance in multimodal datasets, motivating adaptive, region-aware cross-modal fusion.

Multi-Scale Fusion via Diffusion Wavelets

The refined modality-specific graphs are projected into the diffusion wavelet domain to capture both local and global topological nuances. At each diffusion scale—ranging from local high-frequency (semantic boundary) to global low-frequency (semantic backbone)—the response entropy of each modality is measured, providing a data-driven criterion for adaptive modality fusion. This enables CAST to allocate greater weight to the structurally healthier modality at each scale while reconstructing a unified topology that preserves all degrees of semantic granularity.

Distribution Matching and Coreset Optimization

Anchored on the unified latent topology, CAST poses coreset optimization as a continuous proxy selection problem, where proxies are required to align with the multi-scale diffusion wavelet response of the original dataset. The alignment loss function comprises three terms: a Sliced Wasserstein Distance for global distribution matching, an edge-protection term emphasizing coverage of high-energy boundaries, and a frequency-domain coverage loss penalizing omissions in sparse regions. Proxies are mapped back to real samples using the Hungarian algorithm to form the discrete coreset.

Figure 2: The CAST pipeline integrates multi-stage topology construction, adaptive fusion, multi-scale matching, and optimized coverage.

Figure 3: CAST's coreset selection process relies on multi-scale diffusion wavelet matching and local relational coverage for highly-informative, compact subset selection.

Local Soft Relational Coverage (LSRC)

CAST introduces the LSRC mechanism to overcome proxy over-allocation in dense topological regions. LSRC combines direct spatial coverage with propagation over a soft relation graph, which merges geometric proximity and cross-modal topological priors. This ensures that each proxy efficiently covers a broader cluster, suppressing redundancy and incentivizing representative diversity. An explicit loss term encourages indirect coverage sufficiency and neighborhood smoothness, further regularizing the coreset against local mode collapse and redundancy.

Experimental Results

Comprehensive experiments on Flickr30K and MS-COCO reveal that CAST achieves systematically superior bidirectional retrieval performance under severe compression, notably outperforming all state-of-the-art unimodal and multimodal selection and synthesis baselines. The quality of CAST-selected coresets is especially pronounced for text retrieval metrics, which are sensitive to cross-modal semantic preservation. Ablation studies confirm the necessity of each CAST component, with wavelet-domain alignment and LSRC providing particularly large gains.

Figure 4: CAST delivers superior image-text retrieval performance on MS-COCO across all compression budgets, outperforming alternative coreset and synthesis strategies.

Generalization, Efficiency, and Ablations

CAST demonstrates remarkable cross-architecture generalization: coresets selected with CAST transfer more robustly to transformer and ResNet-based backbones compared to synthesis-based methods, which exhibit strong architecture bias. For instruction tuning tasks using LLaVA-1.5-mix-665K and Qwen2-VL-2B, CAST-selected subsets yield consistent, measurable improvements over random selection on a suite of multimodal reasoning benchmarks, even at 1–10% budgets.

In terms of efficiency, CAST requires significantly lower computational resources than synthesis approaches (e.g., LoRS, RepBlend), which involve continuous optimization of synthetic data. This is crucial for sustainable AI development, as CAST can be applied to large-scale datasets with a fraction of the GPU hours and energy.

Ablation studies across coreset subsets, topology fusion, LSRC, and wavelet scales show that all aspects are necessary for maximal retrieval and coverage performance. Overly coarse scales or the absence of LSRC lead to notable drops in mean recall, confirming the importance of CAST's design principles.

Figure 5: CAST shows strong generalization across architectures, high computational efficiency, and clear benefits from each framework component, as shown in ablation results.

Implications and Future Directions

CAST provides a robust, scalable, and energy-efficient paradigm for multimodal coreset selection, substantiated by strong empirical results and analytic ablation. The theoretical implications are twofold: first, it establishes the practical value of fine-grained multi-scale topology and region-specific modality fusion for structural data summarization; second, it demonstrates that local, relation-aware coverage constraints are essential for maintaining coreset diversity under dense sampling regimes.

CAST is immediately applicable to dataset curation and training acceleration in large-scale vision-LLMs, multimodal retrieval, and instruction tuning. Future research could extend CAST to video-language, speech-text, and broader heterogeneous multimodal domains, as well as adaptive parameterization for efficient transfer across vastly different data distributions. Further system-level optimizations—such as accelerated graph construction and large-batch multi-scale signal processing—would support even larger-scale deployments.

Conclusion

CAST addresses the core challenges in multimodal dataset compression: cross-modal information imbalance, multi-scale structural preservation, and redundancy-aware sample selection. It achieves superior retrieval and transfer performance while minimizing energy requirements, providing a rigorous foundation for scalable, sustainable multimodal learning. The methodological innovations in topology fusion and coverage constraints will inform future work on efficient representation learning in multimodal AI.