HyTRec: A Hybrid Temporal-Aware Attention Architecture for Long Behavior Sequential Recommendation

Abstract: Modeling long sequences of user behaviors has emerged as a critical frontier in generative recommendation. However, existing solutions face a dilemma: linear attention mechanisms achieve efficiency at the cost of retrieval precision due to limited state capacity, while softmax attention suffers from prohibitive computational overhead. To address this challenge, we propose HyTRec, a model featuring a Hybrid Attention architecture that explicitly decouples long-term stable preferences from short-term intent spikes. By assigning massive historical sequences to a linear attention branch and reserving a specialized softmax attention branch for recent interactions, our approach restores precise retrieval capabilities within industrial-scale contexts involving ten thousand interactions. To mitigate the lag in capturing rapid interest drifts within the linear layers, we furthermore design Temporal-Aware Delta Network (TADN) to dynamically upweight fresh behavioral signals while effectively suppressing historical noise. Empirical results on industrial-scale datasets confirm the superiority that our model maintains linear inference speed and outperforms strong baselines, notably delivering over 8% improvement in Hit Rate for users with ultra-long sequences with great efficiency.

• The paper introduces a dual-branch hybrid attention model that separates long-term stable preferences from short-term intent to achieve both precision and efficiency.
• It employs a Temporal-Aware Delta Network alongside selective softmax attention to dynamically weight features and adapt to rapid interest shifts in ultra-long sequences.
• Empirical evaluations on Amazon benchmarks demonstrate significant improvements in hit rate, NDCG, AUC, and throughput compared to transformer-based and linear attention models.

HyTRec: A Hybrid Temporal-Aware Attention Architecture for Long Behavior Sequential Recommendation

Motivation and Context

Accurately modeling user preferences in ultra-long behavioral sequences is a core challenge for large-scale generative recommender systems. Existing approaches reveal a persistent trade-off: softmax attention enables semantically precise modeling at quadratic computational complexity, whereas linear attention variants lower inference costs at the expense of retrieval fidelity and insufficient adaptability to rapid interest shifts. The prevalence of industrial scenarios involving sequences with tens of thousands of interactions necessitates architectures that uphold both efficiency and precise intent tracking.

Recent works—from sparse attention schemes to state-space models—have partly addressed these constraints, but suffer from limited injectivity and suboptimal response to recent intent changes. The shift towards leveraging LLM-inspired architectures for sequence recommendation increases the urgency for innovation in scalable, high-fidelity sequence modeling.

HyTRec Architecture

The HyTRec framework explicitly decouples long-term and short-term behavior modeling via a dual-branch hybrid attention structure.

• Sequence Decomposition: The user sequence $S_u$ is partitioned into $S_u^{long}$ (historical, stable preference) and $S_u^{short}$ (recent, intent drift-sensitive). This decomposition enables specialized processing targeting the distinct temporal modes of user behavior.
• Hybrid Attention Stack: The long-term branch comprises predominantly Temporal-Aware Delta Network (TADN) linear attention layers, interleaved at a configurable ratio with standard softmax multi-head self-attention to periodically restore retrieval fidelity and high-capacity dependency tracking.
• Short-Term Precision Modeling: The short-term branch applies full softmax attention exclusively to $S_u^{short}$, preserving semantic integrity for near-term intent.
• Parallel Fusion: Outputs from both branches are fused to compute the next-item probability distribution.

  Figure 1: The overall architecture of HyTRec, explicitly separating historical and short-term modeling branches, with fusion for next-item prediction.

Temporal-Aware Delta Network (TADN)

The TADN module is central to HyTRec's long-term branch, delivering linear complexity while mitigating the standard semantic dilution of long-sequence linear models. Key elements:

• Temporal Decay Gating: Each historical interaction is weighted with a temporal decay factor $$\tau_t = \exp(-\frac{t_{\text{current}} - t_{\text{behavior}^t}}{T})$$ that dynamically quantifies its relevance to the next-item prediction. The gating function combines temporal decay and feature similarity.
• Selective Update: The gating weight $g_t$ controls a convex combination of short-term feature deviations $\Delta \mathbf{h}_t$ and the long-term embedding, ensuring recent high-value signals dominate the output while suppressing noise from distant, irrelevant history.
• Linear State Recurrence: The state update rule reuses the Gated DeltaNet recurrence with temporally-aware decay, resulting in a fused feature representation prioritizing recent information.

The TADN allows HyTRec to dynamically reweight and filter long behavioral sequences, outperforming both standard linear and softmax-only models in adaptability to interest drift.

Empirical Results

Recommendation Performance

On multiple Amazon benchmarks, HyTRec achieves significant improvements across H@500, NDCG@500, and AUC:

• Beauty: H@500 of 0.6643, AUC of 0.8655, exceeding all baselines including transformer-based (SASRec, HSTU) and hybrid models (e.g., Qwen-next).
• Electronics: H@500 of 0.3272, AUC of 0.8760, second only in hit rate to Qwen-next but leading in AUC.
• Movies & TV: H@500 of 0.7070, NDCG@500 of 0.6268, approaching the upper bound of transformer architectures, while maintaining efficiency.

Efficiency and Scaling

HyTRec's throughput remains stable as sequence lengths rise from 100 to 12,000 tokens per user. For sequences of length 5k, HyTRec sustains 65.3K tokens/sec versus HSTU's 28.7K tokens/sec and Transformer’s greater drop-off. This result demonstrates that hybrid attention delivers genuine linear scaling in industrial long-sequence settings—with a minimal penalty incurred by sparse softmax layers.

Figure 2: Training throughput for models at fixed parameter scales on a single V100 GPU with increasing sequence lengths. HyTRec maintains quasi-linear scaling.

Ablation and Component Analysis

Ablation investigations show that both the TADN branch and the short-term softmax attention branch are essential: removing either incurs a substantial drop in H@500, NDCG@500, and AUC. Their integration yields the strongest scores, confirming the necessity of explicit temporal-stratified modeling.

Hybrid Attention Ratio Selection

Varying the ratio of linear to softmax attention layers, the optimal balance (max efficiency/latency trade-off) is empirically found at 3:1—three TADN layers per softmax layer. Ratios with greater sparsity of softmax layers degrade performance, whereas more frequent softmax insertion increases latency with modest gain.

Figure 3: Performance and latency trade-offs as a function of hybrid attention ratio. The 3:1 ratio yields maximal efficiency and recommendation accuracy.

Hyperparameter Sensitivity

The number of attention heads and expert modules were analyzed. Both metrics reveal a sweet spot (2-4 heads and 4 experts) beyond which gains saturate or regress, with increased inference latency as configuration scales.

Figure 4: The impact of expert module count on key metrics and latency. Four experts represent the optimal setting before diminishing returns.

Figure 5: Performance against the number of attention heads, with best trade-offs at 2-4 heads depending on the target metric.

Practical and Theoretical Implications

HyTRec operationalizes hybrid attention for ultra-long sequential recommendation, successfully circumventing the quadratic bottleneck of softmax attention while preserving expressiveness and semantic fidelity. The explicit temporal gating in the TADN aligns with findings from state-space modeling (S4, Mamba), but introduces time-adaptive weighting lacking in existing linear attention stacks.

Practically, this architecture is suitable for deployment in large-scale real-time recommender engines, especially where user histories are extensive and cold-start or silent-user generalization is required. The framework's dual-branch nature is adaptable to future dynamic sequencing tasks and can be extended to cross-domain recommendation tasks, with strong cross-domain transfer results.

From a theoretical standpoint, HyTRec informs the ongoing debate on the semantic capacity of linear versus softmax attention (Han et al., 2024), reinforcing the necessity for periodic high-fidelity retrieval layers in long context modeling.

Future Directions

Key avenues for further advancement include:

• Adaptive Attention Boundaries: Introducing user-adaptive boundaries between short-term and long-term sequence components, personalized by behavioral volatility or inferred stable preference periods.
• Memory-Expanded Hybrid Architectures: Integrating external memory modules to extend capacity beyond the current fixed-state limitation of the TADN, enabling deeper history retention.
• Scenario Generalization: Extending the paradigm to non-e-commerce domains (media, social, cross-lingual) and exploring the structure's robustness to noise and heterogeneity.
• Robustness Modules: Incorporating denoising and noise-aware learning modules to further mitigate challenges with incomplete or noisy historical data.

Conclusion

HyTRec introduces a theoretically grounded, empirically validated solution to efficient and expressive long-sequence modeling for recommendations. Combining a hybrid attention stack with a temporal-aware linear branch, the method demonstrably surpasses prevailing baselines on real-world datasets in both speed and accuracy. The architectural separation of stable preference and transitory intent, together with explicit temporal decay, offers a modular foundation for subsequent advances in scalable generative recommendation.