Three-Branch Sparse Self-Attention

• Three-Branch Sparse Self-Attention is a multi-path architecture that efficiently models long-term sequential user behaviors for CTR prediction.
• It employs personalized, time-aware chunking to segment user actions into variable-length clusters, enabling parallel processing and fine-grained temporal analysis.
• The model fuses global, transition, and short-term attention branches with composite relative temporal encoding to reduce computational cost while improving prediction performance.

A three-branch sparse self-attention mechanism is a structured multi-path self-attention architecture introduced to address the efficiency and personalization challenges in modeling long-term sequential user behaviors, particularly in large-scale click-through rate (CTR) prediction. The approach, as implemented in the SparseCTR model, is designed to capture global interests, transitions between interests, and short-term interests jointly, while offering substantial reduction in computational complexity relative to conventional dense self-attention. This architecture further incorporates personalized, temporally-aware sequence chunking and composite relative temporal encoding, enabling both effective parallelization and fine-grained modeling of user-specific temporal dynamics (Lai et al., 25 Jan 2026).

1. Personalized Time-Aware Chunking

Long user behavior sequences are first segmented into variable-length, user-specific “chunks” based on temporal gaps between actions. Given a sequence $B = \{b_1,\ldots,b_n\}$ with associated timestamps $t_1 \leq \ldots \leq t_n$, adjacent inter-event intervals are computed as $\Delta t_i = t_{i+1} - t_i$. The $|P|$ largest intervals are selected as chunk boundaries, producing $|P|$ variable-length chunks $p_1,\ldots,p_{|P|}$, with a zero-padded chunk $p_0$ if needed. This data-driven, time-aware segmentation respects the natural continuity of user behaviors and ensures that similar event densities are chunked together, which is critical given the personalization and non-stationarity of user logs.

This approach guarantees all users have the same number of chunks, supporting fully-parallel chunk-wise operations for downstream attention mechanisms. The chunking step distinguishes this workflow from other sparse attention approaches that use fixed windows or regular intervals, making it more adaptable to the non-uniform distributions encountered in CTR scenarios.

2. Branch Construction: Three Sparse Self-Attention Paths

The three-branch EvoAttention architecture operates on linearly projected embeddings:

• $Q = E_S W_Q$ (queries from the full sequence including candidate items),
• $K = E_B W_K$ and $V = E_B W_V$ (keys/values from only the behaviors), with $W_Q, W_K, W_V \in \mathbb{R}^{d \times d}$. Each branch specializes in modeling a different temporal or semantic focus:

2.1. Global Interest Branch

Each chunk $p_i$ is aggregated via a multi-layer perceptron (MLP) across its behaviors:

• $k_{p_i} = \mathrm{MLP}(\{k_b \mid b \in p_i\})$,
• $v_{p_i} = \mathrm{MLP}(\{v_b \mid b \in p_i\})$.

Chunk-level keys/values ($K_P, V_P$) form the context for attention, with each query attending to all preceding chunks:

$$\mathrm{Att}_\mathrm{G}(Q, K_P, V_P) = \mathrm{softmax}\left(\frac{Q K_P^\top}{\sqrt{d}}\right) V_P.$$

2.2. Interest Transition Branch

Recent transitions are captured by sampling the $m$ latest behaviors from each chunk, forming $B'$. Projected keys/values $K_{B'}, V_{B'}$ enable attention over recency-weighted behavioral transitions:

$$\mathrm{Att}_\mathrm{T}(Q, K_{B'}, V_{B'}) = \mathrm{softmax}\left(\frac{Q K_{B'}^\top}{\sqrt{d}}\right) V_{B'}.$$

2.3. Short-Term Interest Branch

For each timestep, local context is built from a window of $w$ preceding behaviors plus the compressed user profile $u_c$. This generates K/V matrices $K_{B''}, V_{B''}$, enabling computation of short-term, personalized interactions:

$$\mathrm{Att}_\mathrm{L}(Q, K_{B''}, V_{B''}) = \mathrm{softmax}\left(\frac{Q K_{B''}^\top}{\sqrt{d}}\right) V_{B''}.$$

3. Gated Branch Fusion

Results from all three branches are fused through a gating mechanism. A learnable gating matrix $W_\mathrm{gate} \in \mathbb{R}^{3d \times 3}$ produces attention weights:

$$[\alpha_1, \alpha_2, \alpha_3] = \mathrm{softmax}\left([\mathrm{Att}_\mathrm{G}, \mathrm{Att}_\mathrm{T}, \mathrm{Att}_\mathrm{L}] \; W_\mathrm{gate} \right).$$

The output is a convex combination:

$$\mathrm{Att}(Q, K, V) = \alpha_1 \mathrm{Att}_\mathrm{G} + \alpha_2 \mathrm{Att}_\mathrm{T} + \alpha_3 \mathrm{Att}_\mathrm{L}.$$

Branch fusion is performed per attention head, after which head outputs are concatenated and projected with $W_O \in \mathbb{R}^{d \times d}$.

4. Composite Relative Temporal Encoding

Temporal heterogeneity is modeled by integrating three head-specific bias terms for each head $h$ and position pair $(i, j)$, then inserting the sum into the softmax logits:

• Relative time (log-bucketed):

$$\Delta t_{ij} = |t_i-t_j|,\quad b1_{ij}^{(h)} = -\left\lfloor \log_2(\Delta t_{ij}) \right\rfloor \cdot s_1^{(h)}.$$

• Relative hour (circadian periodicity):

$$H_{ij} = \mathrm{hour\_diff}(t_i, t_j),\quad b2_{ij}^{(h)} = -\sin\left(\pi\, H_{ij}/24\right) \cdot s_2^{(h)}.$$

• Relative weekend:

$$b3_{ij}^{(h)} = \begin{cases} 0, & \mathrm{wk}(t_i) = \mathrm{wk}(t_j) \ -1, & \text{else} \end{cases} \cdot s_3^{(h)}.$$

• Combination: $$\mathrm{bias}_{ij}^{(h)} = b1_{ij}^{(h)} + b2_{ij}^{(h)} + b3_{ij}^{(h)}$$.

The softmax input thus becomes $\frac{QK^T}{\sqrt{d}} + \mathrm{bias}$, encoding inter-event distances, diurnal cycles, and weekly periodicities.

5. Implementation Workflow

The method’s key steps, as presented in layer-level pseudocode, are summarized as follows:

Step	Description	Key Operations/Logic
1	Chunking	Compute $\Delta t$, select top $|P|$ cuts, derive chunks $p_1\ldots p_{|P|}$
2	Projection	$Q = E_S W_Q$, $K = E_B W_K$, $V = E_B W_V$
3	Branch K/V	Aggregate or window $K, V$ for global, transition, and local branches
4	Temporal encoding	Compute bias terms $b1$, $b2$, $b3$ and sum per head
5	Branch attention	Compute $\mathrm{Att}_\mathrm{G}$, $\mathrm{Att}_\mathrm{T}$, $\mathrm{Att}_\mathrm{L}$
6	Fusion	Weighted fusion via $W_\mathrm{gate}$ and softmax
7	Output	Concatenate heads, project via $W_O$

The use of personalized chunking and parallelizable per-chunk computation enables both accuracy and scalability in handling long-range behavior dependencies.

6. Computational Complexity and Scaling Law

The three-branch sparse self-attention dramatically decreases the time and space cost compared to dense self-attention. The floating-point operation (FLOP) count per layer reduces from $O(n^2 d B)$ to

$O(B\,l[n\,|P|\,d + n\,m\,|P|\,d + n\,w\,d])$

where $n =$ sequence length, $B =$ batch size, $l =$ number of layers, $|P| =$ number of chunks, $m =$ transition window size, $w =$ local window size, and typically $|P|, m, w \ll n$.

Parameter overhead is $\mathcal{O}(l d^2)$ due to the chunk-level MLPs, gating, and projection layers.

Empirical evaluation demonstrates that the SparseCTR model using this attention mechanism exhibits a clear scaling law: the AUC improves as a power law in FLOPs, specifically

$\mathrm{AUC}(X) = E - A / X^\alpha,$

with $R^2 \approx 1$. This suggests that doubling the computation reliably increases model performance, a property considered essential for scaling in production settings. In online A/B testing, the model improved CTR by 1.72% and CPM by 1.41% over baselines (Lai et al., 25 Jan 2026).

7. Significance in Recommender Systems and Further Implications

The three-branch sparse self-attention mechanism, as realized in SparseCTR, addresses the distributional complexity and personalization demands of long-term behavior modeling by unifying multiple temporal resolutions and user-centric dynamics. Its use of personalized chunking, multi-branch interaction, and integrated temporal biasing forms a scalable, industrially deployable attention module that differs substantially from standard sparse Transformer variants—reflecting a shift from generic sparse patterns to adaptive, domain-specific attention architectures.

This framework provides a basis for further research into hierarchical and adaptive sparse attention for sequential recommendation, potentially extending to other domains where personalized, temporal sequence structure is crucial (Lai et al., 25 Jan 2026).