HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

Abstract: Most data-mixing methods assume the corpus has already been partitioned into groups, and the choice of those groups determines what a mixer can express. Existing labels, including provenance, topic or format taxonomies, and flat embedding clusters, commit to one semantic axis at one granularity; changing the resolution rebuilds the labels. We argue the bottleneck is the label system, not the mixer, and provide a hierarchical one. HERMES is a data-derived labeling substrate: a Learned Semantic Transform followed by 3-stage residual vector quantization annotates each document once into a coarse-to-fine code whose prefix length controls granularity up to approximately 130k cells. At coarse granularity HERMES sits at a plateau with KMeans-family methods on standard clustering metrics, so the contribution is the substrate, not the clusterer. On 1B-parameter, 25B-token pre-training, the hierarchy exposes an interaction fixed-granularity pipelines cannot test: at one prefix length, a combined Stage-2 rule contrast, equal-subbucket coverage versus size-proportional within-bucket quality top-30%, lifts a 16-task capability macro-average by +0.0253; at the next finer level, the same rule loses its measurable edge as candidate pools contract approximately 5x. HERMES reframes data mixture design from choosing among fixed label sets to navigating a reusable, data-derived granularity hierarchy.

• The paper introduces a hierarchical labeling substrate using a Learned Semantic Transform and three-stage Residual Vector Quantization to enable dynamic corpus partitioning.
• The paper demonstrates that intermediate granularity (L12) yields a statistically significant +0.0253 boost in macro-accuracy via quality-based document selection.
• The paper illustrates that excessively fine-grained partitions lead to candidate competition, emphasizing the need to balance granularity with corpus size.

HERMES: Hierarchical Multi-Granularity Labeling for Pre-training Data Mixtures

Introduction and Motivation

Pre-training data mixtures for LLMs intrinsically depend on how corpora are partitioned for mixing and sampling. Corpus labeling—deciding which documents should be treated as a group—remains a bottleneck: existing label systems (such as provenance, topic, or flat clustering) invariably fix one semantic axis at a single granularity, impairing the expressivity and adaptivity of downstream mixture strategies. This paper introduces HERMES ("Hierarchical Embedding-based RVQ for Mixture Selection"), a hierarchical, data-derived annotation substrate. HERMES uniquely enables multi-granularity grouping through a single annotation step, supporting coarse-to-fine document partitioning on demand, without re-clustering.

HERMES exposes granularity via hierarchical vector quantization codes, which can be read at various prefix lengths, thus offering practitioners unprecedented flexibility over the granularity of data mixing (Figure 1).

Figure 1: Three corpus-control paradigms. HERMES enables multi-granularity sampling via a data-derived hierarchy, unlike fixed-taxonomy or per-sample selection approaches.

Methodology: Learned Semantic Transform and Hierarchical RVQ

HERMES employs a two-stage annotation pipeline:

1. Learned Semantic Transform (LST): A single linear layer with normalization rotates frozen sentence/document embeddings into a space conducive for vector quantization, optimized under structure-preservation, quantization-aware reconstruction, and orthogonality constraints.
2. 3-Stage Residual Vector Quantization (RVQ): In three stages, each of size $K=256$, HERMES assigns discrete codes $(c_1, c_2, c_3)$ to each document. Reading these codes at increasing prefix lengths yields partitions of increasing granularity: $L_1$ (coarse, 256 buckets), $L_{12}$ ($\sim$65k), and $L_{123}$ ($\sim$130k observed).

The trained codebook supports all prefix-based granularities with a single annotation, obviating the need for repeated clustering for each granularity (Figure 2).

Figure 2: HERMES annotation pipeline: frozen encoder → LST → three-stage RVQ, with multi-level codes enabling granularity control.

HERMES annotations, once produced, allow any sampling or weighting scheme—so-called "mixers"—to operate over groupings at any granularity without re-annotation.

Positioning Among Corpus Labeling and Mixing Methods

Traditional corpus labeling systems—provenance (source), distilled taxonomies (e.g., WebOrganizer), or flat unsupervised clusterings (KMeans, RVQ)—all fix both axis and granularity a priori. By contrast, HERMES enables data-derived, hierarchical labels spanning a continuum of granularities. Whereas prior outer-weight/mixer families (Uniform, DoReMi, RegMix, Chameleon) presuppose group labels, HERMES decouples the underlying label system from mixing, enabling flexible, efficient granularity modulation during mixture optimization—a facility not supported by existing approaches.

Experimental Protocol

A 1B-parameter LLaMA-style model is pre-trained for 25B tokens per configuration on a 50M-document corpus, pre-filtered for gross quality. The diverse suite of capability tasks includes 16 benchmarks (spanning science QA, commonsense, language modeling, and string manipulation) with macro-mean accuracy ("Avg.") as the principal metric.

Comparison baselines include:

• Multiple unsupervised clustering methods (KMeans, MiniBatchKMeans, BisectingKMeans, Flat RVQ) at $K=256$.
• Heuristic taxonomies (WebOrganizer Topic/Format). All configurations share the same underlying embeddings and training hyperparameters.

Core Empirical Findings

1. Substrate over Clustering: No L1 Clustering Superiority

At $L_1{=}256$, five distinct clustering methods are statistically indistinguishable both on intrinsic metrics (cosine compactness, entropy, effective cluster count) and on downstream macro-mean accuracy. HERMES matches KMeans up to $0.0002$ in Avg., establishing that any empirical advantage of HERMES must originate from its granularity flexibility, not from superior coarse clustering.

2. Granularity-Dependent Stage-2 Advantage

A central result is observed at $(c_1, c_2, c_3)$0 ($(c_1, c_2, c_3)$165k buckets) under fixed outer weights (DoReMi at $(c_1, c_2, c_3)$2): switching the Stage-2 sampler from max-entropy coverage (equal mass per sub-bucket, all documents eligible) to a quality-based top-30% selection (based on the FineWeb-Edu score within each sub-bucket) yields a statistically significant +0.0253 increase in macro-accuracy (z $(c_1, c_2, c_3)$32.09), with all else held constant.

Figure 3: Granularity arc: macro-mean accuracy (Avg.) across granularities and samplers, highlighting the $(c_1, c_2, c_3)$4 advantage of quality-based selection.

This outcome demonstrates that meaningful additional structure is accessible at intermediate granularity—large enough for within-bucket ranking to be stable, but not so fine as to dilute quality signal.

3. Diminishing Returns at Maximum Granularity

At the finest granularity ($(c_1, c_2, c_3)$5, $(c_1, c_2, c_3)$6130k buckets), the same quality-based Stage-2 sampling yields no measurable advantage over random coverage (+0.0002 Avg.), a collapse attributed to "candidate competition": as sub-bucket sizes decrease (median pool drops from 2,271 to 429), the within-bucket quality ranking ceases to serve as a reliable proxy for global quality, weakening the efficacy of quality filtering.

Figure 4: Distribution of bucket sizes—granularity refinement rapidly contracts sub-bucket pools, with $(c_1, c_2, c_3)$7 dominated by small buckets.

Capacity and Hierarchical Depth

Ablation on the capacity of the coarsest ($(c_1, c_2, c_3)$8) codebook reveals a non-monotonic effect: increasing $(c_1, c_2, c_3)$9 up to $L_1$0 increases performance, then saturates, supporting the design choice for moderate hierarchical depth and reasonable bucket width.

Figure 5: L1 capacity ablation—optimal accuracy at $L_1$1, with diminishing returns beyond that point.

Theoretical and Practical Implications

HERMES reframes data mixture design from a rigid selection among fixed label sets to dynamic navigation of a data-derived granularity hierarchy. It enables systematic study of granularity-sampler interactions and supports efficient, annotation-efficient experimentation across hundreds of thousands of document groups, with a single labeling pass over the corpus.

The observed boundary when quality ranking loses impact (as group sizes dwindle) has practical consequences: corpus size and hierarchy depth must be matched to avoid candidate competition, and the effectiveness of quality-based filtering is granularity-limited.

For practical data pipeline design, HERMES supports multiple outer weighting families, exposes interpretable hierarchical structure (confirmed by both quantitative NMI analysis and qualitative n-gram signatures), and can be extended to deeper/wider hierarchies as corpora grow.

Future Directions

1. Deeper or wider hierarchies: Candidate competition defines a practical upper bound on codebook depth; scaling to larger corpora can support deeper quantization.
2. Multi-level outer weights: Training or optimizing outer weights at multiple prefix levels may further unlock the substrate's flexibility.
3. Soft, intrinsic within-bucket priors: Utilizing quantization errors as priors for sub-bucket weighting, or moving beyond hard-threshold quality filtering, may yield more robust mixture designs.

These directions are directly aligned with the theoretical and structural view advanced in the paper.

Conclusion

HERMES introduces a hierarchical, data-derived labeling substrate that transforms corpus partitioning for pre-training mixtures into a flexible, granularity-controlled process. Its principal contribution is exposing granularity sweeps as a post-annotation decision variable, facilitating systematic discovery of granularity-sampler interactions. The empirical results robustly demonstrate that neither clustering algorithm choice nor ever-finer granularity are universally optimal: the critical regime is at intermediate bucket sizes, where document quality ranking is stable and impactful. This work directly informs future scaling and mixture design strategies for LLM pre-training and opens new avenues for adaptive, semantics-driven data engineering.

---

Reference:

HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures (2607.02266)