Hybrid Granularity Embedding (HGE)

• HGE is a neural embedding paradigm that integrates token, segment, and document-level (or discrete and continuous) features to capture both global context and local details.
• It employs fusion strategies—such as concatenation, learned projection, and self-knowledge distillation—to combine multi-granular features for improved retrieval, reasoning, and anomaly detection.
• Empirical results in multilingual text retrieval, multimodal reasoning, and cybersecurity demonstrate significant performance gains and enhanced interpretability.

Hybrid Granularity Embedding (HGE) is a neural representation paradigm that systematically fuses multiple semantic granularities—such as token, segment, and document levels for text, or discrete and continuous symbol streams for vision—within a single embedding pipeline. Originating in state-of-the-art models for text retrieval, multimodal LLMs, and cybersecurity, HGE aims to harness complementary strengths: global (coarse-grained) context, local (fine-grained) details, and structured aggregation. Implementations of HGE have demonstrated gains in both performance and interpretability across multilingual IR, multimodal reasoning, and robust anomaly detection, as evidenced in M3-Embedding (Chen et al., 2024), MaVEn (Jiang et al., 2024), and WADBERT (Luo et al., 29 Jan 2026).

1. Foundational Principles of Hybrid Granularity Embedding

HGE designs explicitly encode and fuse information at different granularities, motivated by the observation that semantic meaning is distributed heterogeneously across structural levels:

• Coarse granularity: Captures global semantics, such as overall document meaning or high-level visual concepts.
• Fine granularity: Captures local distinctions—critical for detail-oriented tasks, such as identifying obfuscated attacks or reasoning over fine image regions.
• Intermediate/structural granularity: Facilitates aggregation across meaningful partitions, such as segments in text (e.g., paragraphs) or key–value parameters in structured data.

The central principle is to expose these representations simultaneously, then fuse them—either in the embedding space or at a higher decision level—so that downstream tasks can leverage both breadth and detail.

2. Architectures and Embedding Pipelines

a. Large-Scale Text Models: M3-Embedding

M3-Embedding (Chen et al., 2024) extends an XLM-RoBERTa-large encoder to handle up to 8,192 tokens, producing hybrid granularity embeddings using:

• Document-level ([CLS]) pooling for coarse embeddings.
• Segmental multiple-[CLS] pooling: Inserting an extra [CLS] every 256 tokens, averaging the resulting segment [CLS] vectors as a fine-grained embedding.
• Fusion head: Concatenation and learned projection (followed by layer norm) to produce the final hybrid embedding:

$$e_{\text{hge}} = \mathsf{LayerNorm}\left( W_f \cdot [e_d; e_s] + b_f \right)$$

b. Visual-LLMs: MaVEn

MaVEn (Jiang et al., 2024) introduces a dual-stream HGE for multimodal LLMs, encoding each image both as:

• Discrete symbols: Visual tokens representing high-level concepts, merged into the LLM's vocabulary.
• Continuous patch embeddings: Fine-grained ViT-derived features, projected into the LLM space.
• Fusion: Concatenation of reduced continuous tokens (selected by an MLP-based, text-guided selector) and discrete tokens.

Mathematically, given an image $I$,

$$E_{\mathrm{disc}}(I) = [\dot v^1,\ldots,\dot v^{n_d}], \quad E_{\mathrm{cont}}(I) = [v^1,\ldots,v^{n_c}]$$

and the fused input is

$$V_{\mathrm{input}} = [v^{p_1},\ldots,v^{p_{m_c}}, \dot v^1, \ldots, \dot v^{n_d}]$$

where $p_{m_c}$ are indices of retained patches after reduction.

c. Security Applications: WADBERT

WADBERT (Luo et al., 29 Jan 2026) operationalizes HGE for HTTP requests by fusing:

• Character-level Bi-GRU outputs: Capturing fine orthographic/Unicode obfuscations.
• Subword (WordPiece) embeddings: For semantic integrity and compatibility with BERT backends.
• Parameter-level vectors: Each key–value is a separate embedding, pooled via order-agnostic multi-head self-attention.

The final per-token hybrid embedding is

$$E^{(i)} = E_{\mathrm{wp}}^{(i)} + E_{\mathrm{char}}^{(i)}$$

and parameter embeddings are aggregated as

$$f_{\mathrm{payload}} = \frac{1}{n} \sum_{i=1}^n Z_i$$

where $Z$ is the output from multi-head self-attention on parameters.

3. Fusion Strategies and Aggregation Mechanisms

Fusion is central to HGE and is domain-specific:

• Text retrieval: M3-Embedding utilizes a learned fusion head to combine segment and document [CLS] embeddings. Additionally, retrieval utilizes three heads (dense, sparse/lexical, multi-vector) whose scalar scores are summed at inference for robust ranking.
• Vision-language: MaVEn concatenates reduced continuous and discrete visual tokens, leveraging the transformer backbone’s autoregressive machinery for joint reasoning.
• Parameter sets: WADBERT fuses unordered parameter representations via multi-head self-attention with no positional encodings, ensuring permutation invariance.

Fusion at both embedding and decision levels preserves diverse task-relevant semantics, enabling the model to perform dense, sparse, late-interaction, or hybrid retrieval, and to process unordered or structured data effectively.

4. Multi-Stage Training and Self-Knowledge Distillation

HGE models often employ composite objectives and training regimens to calibrate representations across granularities:

• Contrastive InfoNCE losses: Applied to each head or granularity-specific score (e.g., dense, sparse, multi-vector in text retrieval (Chen et al., 2024)).
• Self-distillation: Collects the sum of all head scores as a “teacher” and aligns each head’s distribution to this ensemble, via:

$$\mathcal{L}'_* = -\sum_{p} \tilde{p}(p|q)\,\log \left[ \mathrm{softmax}(s_*/\tau') \right ]$$

This mechanism stabilizes learning and prevents collapse of any single head (notably the sparse head in M3-Embedding).

• Multi-stage visual training: MaVEn applies stage-wise objectives—patch selector pretraining, discrete stream alignment, continuous stream alignment, and final instruction tuning—to harmonize heterogeneous visual and linguistic signals (Jiang et al., 2024).

This hierarchical adaptation ensures each granularity is both internally coherent and aligned with overall task directions.

5. Application Domains and Empirical Results

HGE has demonstrated task-specific benefits in multiple domains, as summarized below.

Model	Domain	Key Task/Metric	HGE Impact
M3-Embedding (Chen et al., 2024)	Multilingual text retrieval	MIRACL nDCG@10, MLDR nDCG@10	Dense+sparse+multi-vector fusion: +2.2 nDCG@10 over dense-only; SOTA long-doc retrieval
MaVEn (Jiang et al., 2024)	Multimodal reasoning	DEMONBench, SEED-Bench, VQAv2	Hybrid (discrete+cont) outperforms each alone; DEMON: +8.85 points over ViT-only
WADBERT (Luo et al., 29 Jan 2026)	Web security	CSIC2010/SR-BH2020 F1-score (%)	HGE: 99.63/99.50 vs. ~0.12% drop for WordPiece-only; interpretability via attention

Ablations confirm that exclusion of any granularity reduces robustness or accuracy. In particular, the removal of self-distillation in M3-Embedding caused sparse head collapse (36.7 vs. 53.9 nDCG), and WADBERT’s hybrid scheme was essential for outlier-heavy datasets.

6. Implementation Recommendations and Design Patterns

Adoption of HGE requires attention to several technical recommendations, consistently validated across domains:

• Pretraining: Base encoders should be further pre-trained (e.g., via RetroMAE) before supervised retrieval (Chen et al., 2024).
• Batching and length optimization: Efficient binning and split-batch scheduling are crucial for scaling HGE pipelines, particularly for long-form or multi-sequence inputs.
• Granularity injection: In text, auxiliary [CLS] tokens can be inserted at arbitrary strides (paragraph, section, etc.), and their embeddings can be pooled and fused for richer downstream signals.
• Permutation invariance: Structured aggregation (e.g., multi-head attention without positional encoding) allows HGE to flexibly accommodate set-structured inputs (Luo et al., 29 Jan 2026).
• Dynamic reduction: For vision, reducing continuous streams based on discrete semantic guidance optimizes efficiency and preserves important details (Jiang et al., 2024).

A plausible implication is that HGE can be extended to arbitrary new modalities or hierarchical structures by designing appropriate feature extractors and fusion heads for each granularity.

7. Interpretability, Limitations, and Future Directions

HGE inherently enhances interpretability, especially via attention mapping at the structural (e.g., parameter or patch) level—enabling, for example, attribution of suspicious payload parameters in cybersecurity (Luo et al., 29 Jan 2026). Empirical limitations include increased computational cost and the need for careful balancing during training (e.g., via self-distillation).

Potential future directions involve: generalizing HGE architectures into more flexible multi-modal frameworks, exploring adaptive fusion mechanisms, and extending to complex document or multi-image reasoning benchmarks. As HGE enables each granularity to contribute meaningfully and transparently, it is positioned as a foundational building block in the next generation of neural representation learning.