Text-Enhanced Facet-Aware Pre-training Module

• The paper introduces a method to disentangle item representations along multiple semantic facets using supervised contrastive learning.
• It employs a frozen text encoder alongside independent facet projection heads to enhance semantic grounding and address cold-start limitations.
• Empirical results on datasets like ML-20m show improved NDCG and Hit@20 metrics compared to traditional ID-based methods.

A Text-Enhanced Facet-Aware Pre-training module is an architectural and algorithmic paradigm for learning item representations that are (i) semantically dense, (ii) robust to cold-start scenarios, and (iii) explicitly disentangled along multiple interpretable semantic axes or facets. The method was introduced to address deficiencies in item representation for recommendation and multi-modal systems, where simplistic ID-based or single-embedding representations are inadequate for capturing an object’s multi-faceted properties—such as movie genres, starring actors, or product brands. The archetype, as instantiated in the FAME+ sequential recommendation framework, combines frozen pre-trained text encoders, facet-aligned projection heads, and a specialized supervised contrastive learning regimen to produce separably aligned sub-embeddings per facet, thus yielding item encodings that are directly compatible with subsequent facet-aware neural architectures (Liu et al., 18 Jan 2026).

1. Motivations and Foundational Objectives

Traditional sequential recommendation and representation learning pipelines often utilize randomly initialized ID embeddings, resulting in poor semantic grounding and severe cold-start limitations, especially for sparsely-interacted items. Direct consumption of item textual metadata via standard LLMs offers no guarantee that interpretable facet structure (such as genres or directors in movies) will be isolatable or aligned with desired downstream architecture (Liu et al., 18 Jan 2026). The core objective of Text-Enhanced Facet-Aware Pre-training is to enforce, at the pre-training stage, a structural disentanglement: for each of $H$ facets anticipated in the downstream architecture, a dedicated subspace is aligned to a specific class label (e.g., genre, director, brand), such that items sharing the same label are pulled together and items with different labels are pushed apart in the corresponding sub-embedding. This is operationalized by producing a concatenated embedding $\mathbf{e}_i' \in \mathbb{R}^D$, partitioned into $H$ $\ell_2$-normalized facet sub-vectors.

2. Architectural Design

The Text-Enhanced Facet-Aware Pre-training module leverages the following sequential pipeline (Liu et al., 18 Jan 2026):

• Frozen Text Encoder: The raw textual metadata $\mathcal{T}_i$ for each item $i$ (comprising titles, descriptions, genres, brands, etc.) is mapped to a dense vector $\mathbf{e}_i^{\textrm{text}} \in \mathbb{R}^{D_T}$ using a frozen BERT (or comparable transformer) encoder.
• Shared Projection: A small, trainable MLP, parametrized by weights $\mathbf{W}_{\mathrm{shared}}$ and bias $\mathbf{b}_{\mathrm{shared}}$, maps the encoder output into the recommendation embedding space $$\mathbf{h}_i = \sigma(\mathbf{W}_{\mathrm{shared}} \mathbf{e}_i^{\textrm{text}} + \mathbf{b}_{\mathrm{shared}}) \in \mathbb{R}^D$$, where $\sigma$ is typically a ReLU nonlinearity.
• Independent Facet Projection Heads: For each facet $h \in \{1, \ldots, H\}$, there is an independent linear head: $$\tilde{\mathbf{z}}_i^{(h)} = \mathbf{W}^{(h)} \mathbf{h}_i + \mathbf{b}^{(h)}$$, followed by $\ell_2$ normalization, $$\mathbf{z}_i^{(h)} = \tilde{\mathbf{z}}_i^{(h)} / \|\tilde{\mathbf{z}}_i^{(h)}\|_2$$. This produces $H$ sub-vectors of dimension $D/H$ each.
• Facet-Concatenated Embedding: The final item embedding for downstream use is $$\mathbf{e}_i' = [\mathbf{z}_i^{(1)} \| \cdots \| \mathbf{z}_i^{(H)}] \in \mathbb{R}^D$$.

This architecture ensures compatibility with multi-head, facet-aware mixture-of-experts models in subsequent recommendation stages.

3. Supervised Contrastive Learning Objective With Alternating Optimization

The central pre-training mechanism is an alternating supervised contrastive learning objective, with separate optimization for each facet (Liu et al., 18 Jan 2026):

• For each facet $h$, let $y_i^{(h)}$ be the discrete facet label for item $i$.
• For a mini-batch $\mathcal{B}$, define the positive set for anchor $i$ as $$P(i) = \{ p \in \mathcal{B}\setminus\{i\} \mid y_p^{(h)} = y_i^{(h)} \}$$, and anchor/negative set $A(i) = \mathcal{B} \setminus \{i\}$.
• The per-facet supervised contrastive loss is

$$\mathcal{L}^{(h)} = \sum_{i\in\mathcal{B}} \frac{-1}{|P(i)|} \sum_{p\in P(i)} \log \frac{\exp\left( \mathbf{z}_i^{(h)} \cdot \mathbf{z}_p^{(h)} / \tau \right) }{ \sum_{a\in A(i)} \exp\left( \mathbf{z}_i^{(h)} \cdot \mathbf{z}_a^{(h)} / \tau \right) }$$

where $\tau$ is the contrastive temperature (typ. $\tau=0.07$).

• Optimization alternates over facets: for each epoch, each $h$ is selected in turn, a stratified $P\times K$ sampler draws batches with $P$ labels and $K$ samples per label, and $\mathcal{L}^{(h)}$ is computed and back-propagated. During optimization for facet $h$, only the shared projection and head $h$ are updated; other heads are kept frozen.

Stabilization heuristics include batch design ensuring $|P(i)|=K-1>0$, and cycling over rare classes to guarantee coverage.

4. Feature Disentanglement and Facet Alignment

This pre-training paradigm directly achieves disentanglement of different semantic facets: the shared semantic vector $\mathbf{h}_i$ is projected into $H$ disjoint subspaces, each aligned with one facet via its supervised contrastive loss. The contrastive loss explicitly pulls together items sharing a facet label $y^{(h)}$, while pushing apart items with different labels, creating representations for which each sub-vector $\mathbf{z}_i^{(h)}$ encodes only information relevant to facet $h$. The concatenation ensures that all facet representations are present and non-interfering, yielding embeddings pre-aligned for use in scoring, gating, or mixture-of-experts downstream modules (Liu et al., 18 Jan 2026).

5. Empirical Evaluation, Ablation Studies, and Impact

Empirical results demonstrate significant performance improvement from both text enhancement and facet-aware pre-training. On the ML-20m movie dataset ($D=128$, $H=2$, facets Genre and Director):

Model	NDCG@20	Change
FAME (random ID init)	0.1513	—
FAME$_{\mathrm{raw}}$ (BERT+MLP)	0.1602	+5.9%
Full FAME+ (Facet contrastive)	0.1608	+6.4%

Across four public datasets, introducing raw text gives a consistent 2–6% lift in Hit@20 and NDCG@20, and facet-aware pre-training yields an additional 0.5–1.0% gain; in some settings (e.g., Sports dataset), facet disentanglement alone improves NDCG@20 by nearly 13% over the strongest prior baseline. Ablations reveal that primary facets (e.g., Genre) provide most of the improvement, with secondary facets (Brand, Director, etc.) offering supplementary but smaller gains. At end of pre-training, item embeddings are not only semantically grounded but also directly useable for multi-head, multi-facet recommender architectures (Liu et al., 18 Jan 2026).

6. Relationships to Broader Facet-Aware and Text-Enhanced Pre-training Paradigms

Text-Enhanced Facet-Aware Pre-training is situated within a broader context of multimodal and facet/disentanglement-aware representation learning. Related work in vision-language pre-training (e.g., scene text detection via vision–language contrastive and MLM losses (Song et al., 2022)), hierarchical contrastive learning on text-attributed hypergraphs with semantic/facet-aware augmentation (Pan et al., 5 Aug 2025), and multi-facet aggregation in vision–LLMs reinforce the importance of coupling text-derived representations to explicit, often structurally induced, semantic axes. The distinguishing properties of the FAME+ approach are its principled alignment of representation partitions to anticipated downstream semantic and architectural requirements, and its explicit supervised contrastive objective per facet, as opposed to agnostic or purely self-supervised variants.

7. Design Considerations and Practical Implementation

Key architectural and training decisions include:

• Use of a frozen text encoder to extract stable semantic features without overfitting.
• Per-facet linear heads with dedicated parameters for facet subspace alignment.
• Alternating optimization schedule with stratified batch sampling for each facet to ensure positive coverage.
• Strict $\ell_2$ normalization of sub-embeddings for contrastive stability.
• Pragmatic hyperparameter choices, such as $P\times K$ batch composition, temperature $\tau=0.07$, and pre-training epochs (e.g., 300 for FAME+).

These choices result in embeddings that are robust to data sparsity and semantic cold-start, yielding performance improvements above conventional or naïve text-initialized alternatives (Liu et al., 18 Jan 2026).

A plausible implication is that similar strategies may further generalize to other multi-facet or multi-modal settings where structurally disentangled embeddings enhance downstream interpretability, transfer, or sample efficiency.