Two-Tower Model Conversion for Recommender Systems

• Two-Tower Model Conversion is a method for adapting dual-encoder architectures to handle diverse recommendation tasks by mapping entities into a shared low-dimensional space.
• It details task-specific adaptations—such as similarity search, complementary suggestions, and inspirational discovery—through modifications in tower inputs and ANN indexing.
• Empirical outcomes from large-scale deployments show measurable gains in metrics like CTR, GMV, and AUC while maintaining computational efficiency and embedding consistency.

A Two-Tower model is a neural architecture for candidate retrieval in large-scale recommender systems. It features two symmetric or asymmetric subnetworks (“towers”) mapping entities (e.g., users and items, or two items) into a shared low-dimensional embedding space. Conversion of standard Two-Tower models enables new use cases or expressive power with minimal changes to model structure, input features, or post-processing logic. This article details both practical and advanced forms of Two-Tower model conversion, grounded in large-scale production deployments and recent research advances.

1. Base Two-Tower Model: Architecture and Training

The canonical Two-Tower (“TT”) retrieval model is a symmetric network, frequently parameter-shared, that embeds “query” and “target” entities into a common $d$-dimensional space via an identical encoder. In e-commerce contexts (e.g., Allegro.com), each product is described by a feature set comprising textual attributes (e.g., tokenized title) and structured categorical attributes (e.g., price bin, category, seller ID). Each categorical feature $f_i$ is embedded by a learnable table $E_i \in \mathbb{R}^{|V_i|\times d_i}$, and the title via token pooling or a compact transformer, yielding a stacked feature vector $x = [f_1; f_2; \ldots; f_M]$.

These are concatenated into $$e_{\mathrm{emb}}(x) = \mathrm{concat}( E_1[f_1], \ldots, E_M[f_M]) \in \mathbb{R}^D$$, processed through $L$-layer fully connected MLPs with ReLU activations, topped by $L_2$ normalization to produce the final normalized embedding $p(x) \in \mathbb{R}^d$:

$$h^{(l+1)} = \mathrm{ReLU}(W^{(l)} h^{(l)} + b^{(l)}), \quad p(x) = \frac{h^{(L)}}{\|h^{(L)}\|_2}$$

The predictive relevance between a query $q$ and candidate $t$ is given by the dot product $s(q, t) = \langle p(q), p(t) \rangle$, proportional to cosine similarity. Training employs sampled-softmax loss with mixed negative sampling, using positive (q, t) pairs from co-viewed history and negatives from “hard” and random sampling ($\ell_{\mathrm{SS}}(q, t) = -\log P(t|q)$) (Osowska-Kurczab et al., 19 Jul 2025).

2. Task-Specific Two-Tower Model Conversion

A principal advantage of the TT architecture is flexible adaptation to diverse recommendation roles by limited changes to either the tower inputs or the serving logic, while reusing the core encoder.

Task	Model Modification	Serving/Index Change
Similarity Search	None (base TT)	Flat ANN search
Complementary	Add category to query	Flat ANN, per-category
Inspirational	No encoder transform	Hierarchical ANN, skip

2.1 Similarity Search (Substitutes): The original model is trained and deployed as described above, returning closest items to a query in embedding space via ANN search.

2.2 Complementary Suggestion (Complements): The query tower is augmented with an additional “complementary category” feature $c^*$ (from a learned one-to-many mapping, e.g., co-purchased goods). The modified query embedding is $$z(q, c^*) = \mathrm{concat}(p_q, E_{\mathrm{cat}}[c^*])$$, passed through the same MLP + $L_2$-norm pipeline to obtain $p_{\mathrm{cmpl}}(q, c^*)$. Training uses a composite loss:

$$\ell(q, t, c^*) = \ell_{\mathrm{SS}}(q, t) + \lambda \ell_{\mathrm{cat}}(q, c^*)$$

where $\ell_{\mathrm{cat}}$ is a category reconstruction log-loss, reinforcing use of the category signal. Serving infers multiple $p_{\mathrm{cmpl}}$ embeddings—one for each candidate category—and retrieves from a common index, interleaving lists to build the carousel.

2.3 Inspirational Discovery (Serendipitous Content): The encoder remains unchanged, but the global ANN index is replaced with a two-level hierarchical IVF (Inverted File) index (using Faiss):

• Offline, catalogue embeddings $p(t)$ are clustered by k-means, producing centroids $\{\mu_k\}$. Each cluster forms its own sub-index.
• At retrieval, recent user-views form multiple “category representatives” $r_j$, each embedded as $p(r_j)$. These probe the top-level IVF for $n$ centroids, skipping the $l$ closest for novelty, with $R$ items drawn from the corresponding clusters and interleaved to increase diversity.

3. ANN Search and Indexing in Model Conversion

The efficiency and scalability of TT model conversion hinge on versatile ANN indexing. In both Similarity-TT and Complementary-TT, a unified flat or IVF-PQ index supports rapid lookups ($\sim$40 ms p99 at 20k rps reported at Allegro (Osowska-Kurczab et al., 19 Jul 2025)), consolidating substitutes and complements retrieval. Inspirational-TT leverages a two-level hierarchical IVF index (K ≈ 10,000 clusters), providing top-level novelty filtering and cluster-wise diversity, with lower-level ANN (flat or PQ) for speed and memory efficiency.

All indices are rebuilt daily to adapt to catalogue churn. Life-cycle cost remains low since each conversion reuses the same product-side embeddings downstream of the shared encoder.

4. Fully Interacted Two-Tower Model (FIT) Conversion for Pre-Ranking

Standard TT models support massive-scale retrieval via decoupled user/item towers but lack explicit cross-tower interaction. The FIT architecture (Xiong et al., 16 Sep 2025) addresses this by integrating early and late interaction modules, notably:

• Meta Query Module (MQM): Learns a meta-matrix $W_{\text{meta}} \in \mathbb{R}^{d \times k}$, projecting raw feature vectors $f_u, f_v$ into a meta space. Early interaction vectors $r_{\text{early}}$ are formed via concatenation, elementwise product, or small MLP on $$q_u = f_u W_{\text{meta}}, e_i = f_v W_{\text{meta}}$$.
• Lightweight Similarity Scorer (LSS): Replaces the final dot-product with a learned scorer over the head-wise projected tower outputs. Subspaces $z_u^h = W^h_u h_u + b^h_u$ are used to compute a similarity matrix, then passed through shallow FCs, flattened, and scored via inner product with parameter $w$.

FIT preserves the computational structure of TT—offlining the item-side heavy computation, minimal additional serving latency, and constant storage cost—while providing both explicit feature-level conditioning (via MQM) and a more expressive similarity scoring (via LSS).

Empirical results demonstrate substantial AUC gains (e.g., +14.3% on ML-1M, +10% on Taobao Ad Click) with negligible additional inference latency (increase of 1.5 ms for 2000 candidates) (Xiong et al., 16 Sep 2025). Implementation involves augmenting two-tower code with $W_{\text{meta}}$, soft-attention training logic, early-interaction input fusion, and the LSS scorer for the candidate set.

5. Empirical Outcomes and Business Impact

Two years of A/B testing at Allegro validate the practical value of TT conversion strategies:

• Similarity-TT (substitutes): On the mobile app, click-through rate (CTR) increased by +2.11% and gross merchandise value (GMV) by +0.13%; desktop CTR +2.37%, GMV +0.29%.
• Complementary-TT (complements): Mobile app CTR +1.62%, GMV +0.09%; desktop CTR +1.06%, GMV +0.31%.
• Inspirational-TT: Desktop, carousel layout click-to-action (CTA) +3.12%, conversion rate (CVR) +1.38%, bounce rate −4.09%; infinite scroll CTA +4.15%, CVR +2.22%, bounce rate −5.74% (all statistically significant at $p<0.01$) (Osowska-Kurczab et al., 19 Jul 2025).

FIT conversion for pre-ranking yields up to +14.3% AUC improvement with equivalent storage and nearly identical inference cost (Xiong et al., 16 Sep 2025).

6. Model Maintenance, Deployment, and Extensions

Both Allegro’s conversion strategy and the FIT framework demonstrate that a single core product/item encoder suffices for a spectrum of recommendation intents—with only lightweight query-tower augmentation or post-tower fusion needed to convert between modes. Daily ANN index rebuilding integrates catalogue updates with no impact on invariance of product embeddings in Similarity- and Complementary-TT. In FIT, early-interaction “meta query” lookups and LSS-based late fusion preserve TT paradigm efficiency.

Recommended implementation steps include (for FIT): adding and initializing the meta-matrix; integrating attentive logic in the feature pipeline; substituting the output scorer; and storing minimal per-item meta indices.

A plausible implication is that the modularity of TT conversion supports continual expansion to new recommendation scenarios (e.g., bundles, diversity, context conditioning) with limited infrastructure cost, while maintaining tight control over online inference budgets. This flexibility, empirically substantiated at industrial scale, underlies the widespread adoption of TT architectures and their conversion extensions across the recommender systems domain (Osowska-Kurczab et al., 19 Jul 2025, Xiong et al., 16 Sep 2025).