Group-aware Multimodal LLMs

Updated 9 February 2026

The paper introduces G-MLLMs that integrate multimodal sensor data with explicit group structures to enhance group-level reasoning and prediction.
It employs group-aware tokenization and hierarchical context modules to process visual and sensor inputs, optimizing inference and interpretability.
Empirical results demonstrate significant improvements in group activity detection and interaction prediction, with notable gains over baseline models.

A Group-aware Multimodal LLM (G-MLLM) is a specialized class of LLMs designed to process, reason about, and predict behaviors that depend not just on multimodal sensor data but also on explicit group structure, group activity, and collective semantics. G-MLLMs integrate sensor streams or visual-linguistic inputs across participants, structure these into hierarchical or grouped forms, and leverage LLM capabilities—often with custom architectural modifications such as group tokenization, context hierarchies, and group-activity supervision—to advance explainable and efficient group-level understanding and prediction. This approach underpins advances in collaborative sensing, group interaction prediction, and group activity detection across both vision- and sensor-centric settings (Romero et al., 18 Nov 2025, Huang et al., 2024, Peng et al., 19 Sep 2025).

1. Architectural Principles and System Pipeline

G-MLLM designs typically extend standard MLLMs through explicit modularization for group reasoning and token grouping. A representative pipeline begins with raw multimodal streams (such as audio, head/hand pose, gaze, and task logs for collaborative mixed reality), which are preprocessed and encoded into modality-specific features: turn-taking for audio, pairwise proximity for position, and shared-attention durations for gaze. Hierarchical context modules further organize these into:

Individual profiles (session-level behavioral clusters)
Group structure (momentary sociogram metrics such as density, reciprocity, centrality)
Temporal context (recent window trends, current phase via VAE clustering) These are serialized into natural language prompts that supply multi-level group context to a base LLM (e.g., Gemma-2B) via zero-shot, few-shot, or finetuned LoRA-based prediction.

In vision-centric G-MLLMs, grouping-based token formation is employed: images (or videos) are first divided into patches, then summarized via learnable semantic group tokens (using attention masks to preserve pretrained features), with only the grouped semantic tokens projected to the LLM for downstream reasoning (Huang et al., 2024, Peng et al., 19 Sep 2025).

2. Group-aware Tokenization and Context Construction

A core innovation in G-MLLMs is group-centric tokenization and context construction. In vision settings, this includes:

Learnable "semantic tokens" initialized alongside patch tokens
Similarity-based patch-to-group assignment via Gumbel-Softmax for discrete clustering
Merging and summarization of patch features into semantic group tokens, dropped from the token stream before LLM inference
Use of isolated attention in initial layers, preventing group tokens from altering patch representations, thus maintaining visual priors

In sensor-driven contexts, context construction is formalized with explicit concatenation:

$C^t = \operatorname{concat}(I_1, \ldots, I_n, S_t, T_{t-\Delta:t})$

where $I_i$ are individual profiles, $S_t$ is the fused group structure, and $T_{t-\Delta:t}$ is the recent temporal window context (Romero et al., 18 Nov 2025).

Instructional prompts may additionally insert explicit activity and group tokens (e.g., <ACT>, <GROUP₁>–<GROUP_K>) into the LLM vocabulary, enabling the decoder to associate sequence representations directly with groupwise semantics (Peng et al., 19 Sep 2025).

3. Modality Encoding and Assignment Algorithms

The encoding stage involves transformation of continuous and structured data streams into temporally resolved, group-aware features.

Sensor modalities are encoded as follows (Romero et al., 18 Nov 2025):

Conversation: $x_i^{t, \text{audio}} \rightarrow c_i^t = f_\text{conv}(x_i^{t, \text{audio}})$ , where $f_\text{conv}$ involves diarization and VAD.
Proximity: Compute $p_{i,j}^t$ via integration of indicator functions over positional closeness.
Shared attention: $a_{i, j}^t$ via temporal overlap of object fixation.

For visual grouping (Huang et al., 2024):

Let $X_\text{img} \in \mathbb{R}^{M \times d}$ be patch embeddings, and $X_\text{sem}\in \mathbb{R}^{N \times d}$ the semantic tokens.
A discrete assignment matrix $\hat{A}$ is obtained using straight-through Gumbel-Softmax over learned similarity projections, assigning each patch to a single group.
Merged group tokens $S_i$ are computed as

$S_i = s_i + W_o \frac{\sum_j \hat{A}_{i,j} (W_v x_j)}{\sum_j \hat{A}_{i,j}}$

enabling efficient and semantically aligned group compression before projection into the LLM.

4. LLM Integration Paradigms and Training Procedures

G-MLLMs leverage multiple LLM integration paradigms:

Zero-shot prompting: Context is passed together with task specification, requiring no in-context examples.
Few-shot in-context learning: A single (or more) example context–prediction pair is prefixed to the input, with example selection based on phase similarity, random, or diversity-driven sampling; retrieval method yields marginal returns over random selection (0.5%).
Supervised fine-tuning: LoRA adapters are placed on projection modules (e.g., $q_\text{proj}$ , $v_\text{proj}$ , $o_\text{proj}$ ), optimized over cross-entropy of explicit group interaction matrices (Romero et al., 18 Nov 2025).

For vision-driven group activity detection, explicit group (<GROUPₖ>) and activity (<ACT>) tokens are introduced and optimized via multi-label classification losses on their hidden representations. Loss components may include:

Per-actor action ( $\mathcal{L}_{ind}$ )
Group activity and membership consistency ( $\mathcal{L}_{group}, \mathcal{L}_{mem}, \mathcal{L}_{con}$ )
Activity multi-label ( $\mathcal{L}_{act}$ ) (Peng et al., 19 Sep 2025).

LLM outputs are parsed to generate group-level sociograms or assign group activity labels in each window.

5. Empirical Performance and Analysis

Experimental evaluation demonstrates that G-MLLMs achieve state-of-the-art performance on group-level reasoning and activity detection:

In collaborative MR group prediction (Romero et al., 18 Nov 2025), context-aware LLMs reach up to 0.96 weighted Jaccard similarity for conversation prediction, representing a 3.2× performance increase over LSTM baselines (0.30), and maintain time-to-first-token latencies below 35 ms.
For proximity prediction, group structure awareness confers an additional 6% gain over baseline.
In simulation (autoregressive) mode, performance suffers catastrophic collapse due to error propagation, with up to 99% conversation degradation within two windows, revealing brittleness in sequential prediction.
On vision-language benchmarks, group-token-based grouping (VisToG) preserves ≥98.1% of baseline accuracy using only 128 tokens while cutting inference by 27% or more. Performance remains robust (PRT > 90%) down to 64 tokens; non-grouping baselines collapse under strong compression (Huang et al., 2024).
In group activity detection, LLMs with explicit group and activity tokens (LIR-GAD) outperform prior approaches (+7.4 mAP on Café, +1.6 mAP on JRDB-Act) and show qualitative interpretability in group splits and activity assignments (Peng et al., 19 Sep 2025).

6. Limitations and Future Directions

Limitations revealed in empirical analysis include:

Shared attention reconstruction is ineffective (0% recall), reflecting severe class imbalance, lack of semantic gaze-object modeling, and the inability of discrete text tokens to capture geometric relations.
Simulation brittleness stems from context drift due to cascading prediction errors, affecting especially conversation sociograms.
Marginal gains from advanced few-shot retrieval or example selection approaches indicate saturation under simple conditions.

Promising future directions span:

Dynamic context adaptation: Modality- and task-specific variable-length histories.
Hybrid modules: Combining LLMs for semantic reasoning with lightweight statistical predictors for robust error handling in simulation.
Constrained decoding: Network-consistent outputs that reinforce group structure properties.
Vision-language fusion: Embedding object-centric semantics into gaze and attention encodings, as in PaLI-style models.
Periodic oracle grounding: Inserting ground-truth context periodically to counteract context drift.
Sensor modality prioritization: Tiered input use (e.g., audio-only for conversation, audio+tracking for proximity, eye+vision for shared attention).

A plausible implication is that the G-MLLM design paradigm generalizes to arbitrary modalities—by constructing group tokens and hierarchical or instruction-driven context, models can unify group semantics, reduce compute, and enable interpretable, robust group reasoning across domains.

G-MLLMs synthesize concepts from:

Multimodal LLMs with projection modules (e.g., LLaVA, LLaVA-Phi-3-V)
Token grouping (VisToG), leveraging pre-trained vision backbones, with isolated attention for efficient group summary and preservation of pre-trained distributions (Huang et al., 2024)
Hierarchical context and group-sociogram modeling via natural language serialization (Romero et al., 18 Nov 2025)
Structured vocabulary extension and activity-specific token learning (Peng et al., 19 Sep 2025)

These approaches mark a shift from conventional unimodal and unguided transformers to systems that are inherently group- and structure-aware, unlocking new parameters for efficient, scalable, and semantically rich models of collective behavior.