Vision-Language Sample Aggregation

• The paper introduces sample aggregation strategies that merge multiple visual–text pairs to enhance multimodal fusion and reasoning.
• It details diverse methods—including concatenation, attention-based reduction, and EM clustering—to address semantic gaps and redundancy.
• Empirical results demonstrate significant gains in retrieval, segmentation, and multi-image reasoning across various benchmarks.

A Vision-LLM with Sample Aggregation encompasses architectures and algorithms that explicitly combine or compress information from multiple visual–text pairs, images, sequences, or probes into unified representations before, during, or after multimodal fusion. Approaches range from concatenating input samples into pseudo-sequences, EM-driven cluster aggregation of features, attention-based reduction, probabilistic marginalization over probes, token-level fusion across depth, and instruction-aware filtering. These procedures target diverse scenarios: video-paragraph modeling from image–text corpora, robust segmentation under semantic gaps, multi-view object labeling, model explainability, and general-purpose, multi-image, and multi-task pipelines.

1. Foundations and Motivation for Sample Aggregation

Sample aggregation in vision-LLMs is motivated by three broad challenges: (i) insufficient data for video–text pretraining, (ii) the semantic dispersion between language prompts and fine-grained visual features, and (iii) the inefficiency or redundancy of per-sample/region strategies when handling multi-image, multi-view, or long-context tasks. For example, most vision-language foundation models trained on image–text pairs lack joint modeling of temporal/event-level correlations, as in the COSA framework, which addresses this by concatenating multiple samples into pseudo-long-form video–paragraph sequences (Chen et al., 2023). In robust medical segmentation, the aggregation paradigm (Expectation-Maximization over embeddings) reduces semantic gap and feature dispersion between domain-abstract language and detailed visual cues (Yu et al., 10 Sep 2025). In object annotation, aggregation avoids hallucination traps in view-wise caption fusion by properly marginalizing VLM outputs across probes (Kabra et al., 2023).

2. Concatenative and Sequence-Based Sample Aggregation

COSA (Concatenated-Sample Pretrained Vision-Language Foundation Model) exemplifies direct aggregation via sample concatenation (Chen et al., 2023). At each training iteration, a group of $K$ image–text pairs is drawn, and both the images and their respective texts are concatenated into longer pseudo-sequences:

• Visual sequence:

$$V = [\,\phi_v(I_1) + E^{temp}_1;~\ldots;~\phi_v(I_K) + E^{temp}_K\,] \in \mathbb{R}^{(K \cdot L) \times d}$$

• Text sequence:

$$S = [\,w_1^{(1)},\ldots,\text{SEP},\ldots, w_1^{(K)},\ldots, w_{N_K}^{(K)}\,]$$

Each sequence is processed with appropriate temporal or positional encoding. This approach synthesizes rich scene/event correspondences and enables transfer of temporal structure without requiring video–text pairs. Ablations reveal that the aggregation—rather than merely increased input length—drives large (>20%) absolute gains in retrieval and multi-step reasoning tasks, with benefits persisting across architectures and datasets.

In VistaLLM, multi-image aggregation is achieved via an instruction-guided image tokenizer based on BLIP-2’s QFormer. Each image is encoded, then L learnable queries (shared across images and conditioned on the instruction prompt) aggregate discriminative visual and textual context, yielding a task-aligned token sequence for each image that is then concatenated for downstream fusion (Pramanick et al., 2023). This enables information-efficient multi-image reasoning, as the LLM receives a compressed, context-filtered set of visual tokens.

3. Attention-Based and Probabilistic Feature Aggregation

Attention-based reduction functions provide fully-learnable, distribution-aware aggregation from sets of regions or words to sequence-level representations. The ScoreAttention mechanism (Stefanini et al., 2020) applies variants of cross-attention to obtain per-element scores, then aggregates features with softmax-derived weights:

• For visual tokens $X$ conditioned on text tokens $Z$:

$Y(X, Z) = \sum_{i=1}^{n_q} S_i(X, Z) x_i$

where $S_i(X, Z)$ are normalized relevance scores. This technique consistently improves over mean/max pooling and CLS-token reduction in VQA and retrieval.

Probabilistic probe aggregation, as in SBMPA for 3D object annotation (Kabra et al., 2023), treats each response as a "vote" with log-likelihood from the underlying VLM. For $I$ probes and $J$ generations per probe:

• Deduplicate each probe’s outputs, keep highest log-score for each canonical response.
• Aggregate across probes using LogSumExp:

$s_{agg}(r) = \log \sum_{i=1}^I \exp(s_i(r))$

• Output the normalized probability $p(r)$ for response $r$.

This procedure robustly handles conflicting or uncertain multi-view predictions, providing both point estimates and full output distributions. It outperforms naïve concatenation and LLM-based text summarization in object type and material recognition.

4. Clustering and EM-Based Semantic Aggregation

The semantic gap between abstract language and domain-specific visual features is directly addressed using Expectation-Maximization (EM) aggregation (Yu et al., 10 Sep 2025). Both vision and text token embeddings are clustered into semantic centers via EM:

• E-step (soft assignment):

$$z_{nk} = \frac{\exp({\mathbf{x}_n \cdot \bm{\mu}_k}/\tau)}{\sum_{j=1}^K \exp({\mathbf{x}_n \cdot \bm{\mu}_j}/\tau)}$$

• M-step (update prototypes, reconstruct features):

$$\bm{\mu}_k^{new} = \frac{\sum_n z_{nk} \mathbf{x}_n}{\sum_n z_{nk}}$$

$$\hat{\mathbf{x}}_n = \sum_k z_{nk} \bm{\mu}_k^{new}$$

These aggregated prototypes replace or supplement patch/token features, summarizing dispersed information and improving cross-modal alignment. In the medical image segmentation context, this approach yields +5–10 Dice-point gains in domain shift robustness relative to both vision-only and prior text-guided baselines.

5. Token Aggregation, Compression, and Redundancy in Deep VLMs

In extremely long-context settings, such as video understanding with 10,000+ frames, token-level aggregation and pruning become critical (Xu et al., 20 Nov 2025). The TimeViper model introduces TransV: at two key depths in a hybrid Mamba-Transformer backbone, salient vision tokens are fused into text/instruction tokens by attention, then redundant vision tokens are permanently pruned:

• At layer $\ell$, keep a subset of vision tokens using either uniform dropping or attention-guided selection.
• Apply cross-attention:

$$\tilde X_1^{\ell} = \mathrm{CrossAttn}_\ell(X_1^\ell,\, \mathrm{TD}_\ell(X_0^\ell))$$

$$X_1^{\ell+1} = X_1^\ell + \tanh(\alpha_\ell)\, \tilde X_1^\ell$$

• Reduced context length enables efficient forward and decoding, with only minimal accuracy loss.

Layerwise analyses demonstrate that vision-to-text information flow causes instruction/text tokens to accumulate nearly all relevant visual knowledge, justifying aggressive token aggregation and redundancy elimination. This phenomenon is unique to hybrid state-space/attention models, as pure Transformer models lose almost all attention mass to vision after shallow layers.

6. Aggregation for Explainability and Model-Level Summarization

Sample aggregation provides rigorous global explainability in vision models. The framework of (Nguyen et al., 27 Aug 2025) applies a VLM explainer (such as GPT-4o-mini or Gemini-1.5-flash) to masked CAM outputs for each sample, scoring them along interpretability dimensions. Aggregation over the dataset yields confusion-matrix stratifications (Correct–High, Correct–Low, Wrong–High, Wrong–Low attention), global error metrics, and bias trend indicators. This approach supports automated diagnosis of systematic error and model behavior drift and detects anomalous class-wise failure rates. Correlation between VLM-based aggregate scores and human acceptability metrics (Pearson $r$ up to 0.64, AR up to 85.6%) substantiates the reliability of this explainability-by-aggregation pipeline.

7. Empirical Effects and Benchmark Impact

Rigorous ablations and comparisons confirm the effectiveness of sample aggregation. In COSA (Chen et al., 2023), concatenated-sample pretraining delivers up to 20 percentage points improvement in text-to-video retrieval (DiDeMo, R@1: 57.8% vs. 37%), measurable gains on Video QA (MSRVTT-QA, +1.7–5 pp), captioning (up to +16.7 CIDEr), and persistent benefits under different vision backbones and corpora. VistaLLM (Pramanick et al., 2023) achieves consistent state-of-the-art on 15+ benchmarks, with its aggregation mechanisms yielding up to +10 CIDEr on COCO captioning and substantial increases in segmentation and multi-image reasoning metrics. EM-based clustering aggregation (Yu et al., 10 Sep 2025) gives 2–10% absolute Dice-score improvements on cardiac and fundus tasks.

In 3D annotation (Kabra et al., 2023), score-based aggregation over views yields an increase in top-1 type inference accuracy (PaLI VQA: 26% vs. CAP3D 22%, Tags 6%), as well as full probabilistic uncertainty. TimeViper (Xu et al., 20 Nov 2025) supports hour-long videos with minimal accuracy drop from token aggregation and internal fusion, improving computational throughput by 40.1% over standard LLMs at scale.

---

Collectively, these results and methodologies establish sample aggregation as a cornerstone for robust, efficient, and generalizable vision-language modeling across data types, domains, and modeling objectives.