ExoFormer: Exogenous Anchor Transformers

• ExoFormer is a Transformer variant that decouples early-layer residual anchoring from deep computational refinement using fixed exogenous projections.
• It employs a unified normalized mixing framework across query, key, value, and gate pathways, enabling independent specialization and significantly reducing attention sink.
• Empirical results demonstrate that Dynamic ExoFormer improves accuracy by 2.13% and achieves baseline loss with 1.84× fewer tokens compared to standard models.

ExoFormer is a Transformer variant built on a fundamental insight: the architectural tension between using early-layer attention projections as residual anchors and refining them through deep progressive computation. Instead of relying on the first layer both as a stable reference and as a computation block, ExoFormer introduces exogenous anchor projections—fixed, learned mappings external to the sequential layer stack—enabling independent specialization in identity preservation and computational refinement. This decoupling is operationalized by a normalized mixing framework applied uniformly across query, key, value, and gate pathways, yielding superior accuracy, data efficiency, and mitigated attention sink. Paradoxical representation collapse observed in ExoFormer variants is explained by the Offloading Hypothesis, wherein external anchors preserve token identity and allow the layer stack to focus exclusively on computation (Su, 13 Jan 2026).

1. Architectural Motivation and Decoupling Tension

Standard Transformer architectures propagate early representations via residual pathways, commonly utilizing the first layer’s attention projections as reusable anchors throughout the network. This internal anchoring forces the initial projections to simultaneously act as (a) a stable feature library and (b) a substrate for progressive refinement, leading to suboptimal trade-offs in stability and expressiveness.

ExoFormer addresses this core tension by constructing anchor projections externally:

• Four dedicated exogenous matrices $(W^a_q, W^a_k, W^a_v, W^a_g)$ act on the input embedding $H_0$ to yield anchor queries, keys, values, and gating logits.
• These anchors are structurally decoupled from all sequential layers, liberating internal representations from the persistent burden of feature preservation.
• As a result, anchor matrices specialize in static token identity, while the main layer stack specializes exclusively in computational refinement.

2. Unified Normalized Mixing Framework

The functional interaction between layer-specific and anchor projections is orchestrated by a normalized mixing mechanism applied to each attention pathway $S \in \{Q, K, V, G\}$ at each layer $n$:

• Current-layer projection: $S_n = H_{n-1} W_n^s$
• Exogenous anchor projection: $S_\mathrm{anc} = H_0 W^a_s$
• Anchor projections are first RMS-normalized to address inter-distributional mismatch.
• Mixed projections: $$\widehat{S}_n = \lambda_{n,1}^s \odot \mathrm{RMSNorm}(S_\mathrm{anc}) + \lambda_{n,2}^s \odot S_n$$

Mixing coefficients $\lambda_{n,1}^s, \lambda_{n,2}^s$ are learnable and initialized to $0.5$, with granularity modes:

• Scalar: single coefficient per component/layer
• Headwise: per-attention-head granularity
• Elementwise: fine-grained per-channel mixing, yielding best static performance

The canonical mixing form (using $\alpha^{(q)}$ for queries, etc.) is:

$$M_q = \alpha^{(q)} \odot Q_\mathrm{internal} + (1 - \alpha^{(q)}) \odot Q_\mathrm{exo}$$

with analogous forms for keys, values, and gates.

3. ExoFormer Variants

Three principal model variants instantiate the ExoFormer idea:

• Static ExoFormer: Mixing coefficients are learned during training but fixed thereafter. Elementwise mixing yields optimal static results.
• NuResFormer (Internal Anchor Baseline): Employs the identical mixing scheme but anchors to the first layer’s projections ($Q_1, K_1, V_1, G_1$), serving as a direct comparator.
• Dynamic ExoFormer: Augments static mixing by producing context-dependent scaling factors per layer with a small MLP, outputting eight context-modulated coefficients ($\gamma_n \in \mathbb{R}^8$) corresponding to each mixing role. The final mix at each step multiplies static $\lambda$ coefficients with dynamic $\gamma$ modulators.

4. Empirical Findings and Performance Metrics

Models utilizing exogenous anchors (ExoFormer) demonstrate consistent empirical superiority over internal anchor baselines across multiple metrics. Key results (using 450M-parameter models, 10B tokens from FineWeb-Edu) include:

Model	Accuracy (%)	PPL	Tokens for Baseline Loss	Attention Sink
Base Transformer	48.14	14.79	10B	0.2572
Gated Attention	48.80	14.64	—	0.0091
ResFormer	49.65	14.32	—	0.0212
NuResFormer (elementwise)	49.68	14.15	—	0.0112
Static ExoFormer	49.85	—	—	0.0041
Dynamic ExoFormer	50.27	14.09	5.43B ($1.84\times$ efficiency)	0.0073

Notably, Dynamic ExoFormer achieves a 2.13-point accuracy improvement over baseline and matches baseline validation loss using $1.84\times$ fewer tokens. Attention sink (the tendency for tokens to disproportionately attend to the first token) is reduced by a factor of 2 compared to Gated Attention and $\sim$60$\times$ compared to baseline (Su, 13 Jan 2026).

5. Representation Collapse and the Offloading Hypothesis

A counterintuitive feature of ExoFormer is the emergence of representation collapse:

• Token features within ExoFormer layers exhibit $>$93–95% cosine similarity, approaching indistinguishability.
• Intrinsic dimensionality of the final layer is suppressed to 738 features (716 in dynamic mode) versus 835 for baseline.
• Deep layers achieve orthogonality to input embeddings, indicating radical interior transformation.

Despite typical negative associations with over-smoothing, ExoFormer’s collapse is empirically benign, explained by the Offloading Hypothesis:

• The exogenous anchor persistently injects high-fidelity token identity into every layer.
• Sequential layers are permitted to “collapse” (purify/compress) their representations, unconstrained by identity preservation.
• Thus, identity is statically sourced from the anchor, while computation is dynamically offloaded to internal layers.

6. Comparative Analysis and Architectural Trade-offs

ExoFormer delivers favorable trade-offs when benchmarked against canonical approaches:

• Standard Transformer: No cross-layer mixing, suffering severe compression valleys and high attention sinks.
• Gated Attention: Reduces sink magnitude via gating but lacks cross-layer residualization.
• ResFormer: Residualizes values alone, with improved perplexity but instability if extended to queries/keys.
• NuResFormer: Implements full gating with internal anchors, stabilizing mixing but unable to resolve first-layer tension.
• ExoFormer: By external anchoring, delivers optimal stability, lowest perplexity, and highest empirical accuracy.

Operational compromises include a moderate increase (+3 million parameters) for anchor matrices and slight training overhead for dynamic mixing. Internal representation smoothing, while pronounced, is functionally offloaded and thus not detrimental.

7. Implementation Overview

The forward computation can be summarized as follows:

H = H0 for n in range(1, N+1): Qn = H @ Wn_Q Kn = H @ Wn_K Vn = H @ Wn_V Gn = H @ Wn_G Qhat = lambda1_Q[n] * RMSNorm(Q_exo) + lambda2_Q[n] * Qn Khat = lambda1_K[n] * RMSNorm(K_exo) + lambda2_K[n] * Kn Vhat = lambda1_V[n] * RMSNorm(V_exo) + lambda2_V[n] * Vn Ghat = lambda1_G[n] * RMSNorm(G_exo) + lambda2_G[n] * Gn Qp = QKNorm(RoPE(Qhat)) Kp = QKNorm(RoPE(Khat)) A = softmax(Qp @ Kp.T / sqrt(dk)) U = A @ Vhat U_tilde = U * sigmoid(Ghat) O = U_tilde @ Wn_O H = H + O H = H + FFN(H)

The code and pre-trained models are publicly available at https://github.com/jon123boss/ExoFormer (Su, 13 Jan 2026).

8. Concluding Perspective

ExoFormer establishes that architectural decoupling—externalizing the anchor projections—enables improved cross-layer information flow and performance. Downstream accuracy is enhanced (+2.13% for dynamic mixing), data efficiency is increased ($1.84\times$ fewer tokens to baseline loss), and attention sinks are minimized. The observed representation collapse is recast as a constructive phenomenon, a direct consequence of functional offloading between identity-anchoring and computational layers.