Positional Decay Reweighting (PDR)

• Positional Decay Reweighting (PDR) is a framework that applies trainable, monotonically decaying weights in sequence models to modulate token influence based on their positions.
• PDR implementations range from scalar to vector parameterizations, with empirical evidence showing superior performance in terms of perplexity and information retrieval.
• The approach extends to applications like linear attention, relative positional encoding, and score aggregation, consistently improving signal extraction and detection metrics.

Positional Decay Reweighting (PDR) refers to a family of mechanisms that introduce monotonically decaying factors or weights into sequence models, allowing for explicit control over the influence of positional distance, historical state, or token-level scores. These techniques impose or learn priors that typically prioritize locality or amplify signal saliency according to sequence position. The PDR principle is foundational both in linear complexity attention models—where decays are multiplicatively applied to recurrent state vectors—and in post-hoc analyses or downstream meta-tasks, where scores are reweighted to maximize information utility or detection performance. Across domains, PDR generalizes previous approaches involving static or heuristic decay kernels, instead supporting learned, context-dependent, or plug-and-play decay curves.

1. Mechanistic Foundations of PDR in Sequence Models

Positional Decay Reweighting is most formally instantiated in linear attention mechanisms. Here, the core recurrence is modified to include a position-dependent decay vector $\lambda_t$:

$$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$

where $s_t$ is the cumulative state, $k_t$ and $q_t$ are the key and query, $v_t$ is the value, and $\lambda_t \in (0,1)^{d/h}$ applies elementwise decay. PDR generalizes static decays (e.g., the fixed scalar in RetNet, the exponential schedule in TNL) by allowing $\lambda_t$ to be learned or computed via a function of the input or position, thereby imposing a trainable locality prior: recent tokens are given higher weight, and information from distant tokens is systematically downweighted (Qin et al., 5 Sep 2025).

This mechanism extends beyond linear attention. In transformer-based architectures with relative position encodings, PDR manifests as explicit scaling factors applied either to the positional bias (as in RoPE scaling) or to token importance in loss aggregation.

2. Parameterization and Design Space

The parameterization of the decay factor $\lambda_t$ spans scalar, vector, and learned functional forms:

• Scalar decay: A single value per head or head-layer, e.g., $\lambda_t^j \in \mathbb{R}$.
• Vector decay: A per-dimension vector, $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$0, supporting finer-grained modulation.

Representative parameterization strategies include:

• Mamba2: $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$1, with $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$2 as per-head trainable parameters.
• GLA, Hgrn2, LightNet, Simple Decay: Various vector or scalar transformations, e.g., sigmoid or exponential mappings of pre-activations, with tunable initial medians and slopes.
• TNL (fixed/learnable scalar): Layerwise exponential or sigmoid-based schedules (Qin et al., 5 Sep 2025).

All pre-activation weights and ancillary parameters are optimized with standard backpropagation through the recurrence. Empirical results demonstrate that, under identical parameterizations, vector decay yields consistently superior performance (e.g., Mamba2-vector achieves PPL ≈ 24.0 vs. Mamba2-scalar ≈ 25.8 on 1.45B models), though scalar schemes can be competitive if highly tuned.

3. Parameter Sharing and Structural Regularity

The computation of $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$3 can involve sharing or reusing parameters across heads or layers:

• No sharing (fully independent): Each head receives its own decay parameterization.
• Partial sharing: Decay weights share computation with key projections (e.g., $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$4 in Hgrn2), or across heads (LightNet).

Uncontrolled sharing often degrades performance by driving decay medians to degeneracy (towards 0 or 1), adversely impacting perplexity and generalization. Robust parameterizations (e.g., Mamba2, Hgrn2) can accommodate shared structures without performance loss, but ad hoc sharing is generally discouraged except when empirically validated (Qin et al., 5 Sep 2025).

4. PDR in Relative Positional Encoding and Scaling

Rotary Position Embedding (RoPE) and similar schemes apply relative rotations to queries and keys, with positional similarity decaying as sequence distance increases. PDR is incorporated here by scaling the decay curve via learned, layer-specific factors:

• Layer-specific scaling: Each transformer layer $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$5 applies a distinct scaling $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$6 to its positional bias, typically learned via a Bézier-curve-constrained genetic search. This adjusts the effective decay rate, slowing it in early/mid layers to recover "middle-context" information and accelerating it near the output for tail focus (Wang et al., 6 Mar 2025).

The core operation rewrites positional embedding as $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$7, so relative attention decays more slowly when $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$8. Layerwise scaling of decay (a direct form of PDR) recovers lost long-range dependencies and balances locality versus global aggregation.

Empirically, uniform scaling only shifts the retrieval bias, while adaptive, learned scaling profiles both recover middle-context retrieval and preserve head/tail salience—the desired trade-off imposed by flexible PDR.

5. PDR as a Plug-and-Play Framework for Score Aggregation

In non-parametric or black-box settings, especially for pre-training data detection and membership inference, PDR reframes aggregation of per-token scores. Theoretical analysis reveals that memorization signals—manifested as unusually high predicted probabilities—are concentrated in high-entropy initial tokens and decay as more context accumulates (Liu et al., 11 Jan 2026). Standard methods average scores uniformly across tokens, potentially diluting early strong signals with late, low-entropy noise.

PDR corrects this by introducing position-dependent weights $$s_t = \operatorname{diag}(\lambda_t)\, s_{t-1} + k_t v_t^\top, \qquad o_t = q_t^\top s_t,$$9 when aggregating token-level scores $s_t$0:

• Linear decay: $s_t$1
• Exponential decay: $s_t$2
• Polynomial decay: $s_t$3

Final detection or inference scores are then

$s_t$4

where $s_t$5 is the aggregation subset (e.g., all tokens or lowest $s_t$6). The procedure is training-free and acts as a post-processing wrapper that boosts detection performance by amplifying earlier, more informative signal regions (Liu et al., 11 Jan 2026).

6. Empirical Outcomes and Best Practices

Empirical studies across language modeling and data detection tasks consistently validate PDR's efficacy:

• Linear Attention: Vector decay parameterizations (e.g., Mamba2, Simple Decay with $s_t$7) achieve superior perplexity and generalization. Optimal median decay per layer is $s_t$8–$s_t$9. Scalar decays must be highly tuned to compete; parameter sharing should be restricted (Qin et al., 5 Sep 2025).
• Relative Position Encoding: Layer-specific scaling recovers "lost-in-the-middle" performance in transformers, with middle-context retrieval accuracy jumping from 55.5% (baseline) to 95.0% after PDR-based scaling (Wang et al., 6 Mar 2025).
• Data Detection: Plug-and-play PDR yields consistent AUROC improvements (e.g., Min-$k_t$0++ AUROC increases from 73.6 to 75.9 for $k_t$1 on WikiMIA; paraphrased setting gains +4.7 points) (Liu et al., 11 Jan 2026). PDR is particularly effective on longer sequences and robust to hyperparameter changes.

Recommended default practices include:

• Using vector PDR with robust parameterizations (Mamba2, Simple Decay).
• Tuning polynomial or exponential decay weights for outlier-based detectors.
• For transformer models, learning layerwise decay profiles via constrained optimization (e.g., Bézier-curve genetic search).
• Avoiding RoPE or similar relative positional encodings in models where the decay parameter $k_t$2, unless careful tuning identifies benefit.

7. Limitations and Scope

While PDR is broadly applicable and consistently beneficial in the tested domains, some constraints are noted:

• Tested primarily on English language corpora; generality for other languages or tokenizations is unproven (Liu et al., 11 Jan 2026).
• Effectiveness may be mitigated if influential tokens are distributed non-monotonically (i.e., memorization is not concentrated at the sequence head).
• Empirical gains from layerwise scaling depend on careful architectural tuning and may interact with other forms of regularization or parameter sharing.
• In relative attention models, RoPE and PDR may trade off, with benefits from RoPE only apparent in near-no-decay regimes ($k_t$3).

The PDR paradigm unifies a broad range of positional decay manipulations under a common information-theoretic and algorithmic framework, supporting robust locality priors, improved signal extraction, and adaptive position-aware computation in autoregressive and attention-based sequence models (Qin et al., 5 Sep 2025, Wang et al., 6 Mar 2025, Liu et al., 11 Jan 2026).