HyperMLP: An Integrated Perspective for Sequence Modeling

Abstract: Self-attention is often viewed as probabilistic query-key lookup, motivating designs that preserve normalized attention scores and fixed positional semantics. We advocate a simpler and more unified perspective: an autoregressive attention head can be viewed as a dynamic two-layer MLP whose weights are instantiated from the context history. From this view, attention scores form an ever-growing hidden representation, and standard MLP activations such as ReLU or GLU naturally implement input-conditioned selection over a context-dependent memory pool rather than a probability distribution. Based on this formulation, we introduce HyperMLP and HyperGLU, which learn dynamic mixing in both feature space and sequence space, using a reverse-offset (lag) layout to align temporal mixing with autoregressive semantics. We provide theoretical characterizations of the expressivity and implications of this structure, and empirically show that HyperMLP/HyperGLU consistently outperform strong softmax-attention baselines under matched parameter budgets.

• The paper proposes reconceptualizing autoregressive attention as dynamic two-layer MLPs that replace softmax normalization with context-conditioned feature mixing.
• It demonstrates that explicit temporal and feature mixing significantly boosts model expressivity and improves performance on language benchmarks.
• The study provides theoretical and empirical evidence that preserving readout rank and employing lag layouts are crucial for maintaining autoregressive consistency and optimal routing.

HyperMLP: Reconceptualizing Autoregressive Attention as Dynamic Two-Layer MLPs

Unified View of Autoregressive Attention

The HyperMLP framework proposes a reconceptualization of self-attention in sequence models. Departing from the traditional probability simplex and softmax-based interpretation, the authors demonstrate that an autoregressive attention head can be equivalently formulated as a context-conditioned, dynamic two-layer MLP whose hidden width grows with the past context length. In this formulation, the "attention scores" are treated not as normalized positional weights but as hidden activations over a dynamic context-dependent representation. The classical queries, keys, and values are thereby demoted to intermediate calculations within a learned feature and sequence mixing operator.

This unified "attention-as-MLP" perspective unlocks a seamless integration of feature and temporal (sequence) mixing, leveraging familiar nonlinearities such as ReLU and GLU in lieu of softmax. The result is a model that approaches context summarization and routing as a composite of input-conditioned subnetwork selection and context-wide gated memory access, circumventing the probabilistic constraints historically attached to attention.

Theoretical Analysis of Expressivity and Routing Geometry

A major theoretical contribution of the paper is the rigorous characterization of the functional class induced by this dynamic two-layer MLP view. The authors show that when the sequence mixing matrices are static, the resulting gating volumes correspond to intersections of polyhedral regions, much like conventional ReLU MLPs. However, when these mixing operators are instead input-dependent and low-rank, the gating geometry becomes a smooth, warped partition of the input space, strictly generalizing the polyhedral case. This warped routing unlocks representation of functions whose decision boundaries cannot be captured by any finite union of hyperplanes—a radical expansion over classical attention operator function classes.

Moreover, the analysis justifies the use of context length in an ever-growing hidden dimension: each autoregressive step adds further width to the dynamic memory pool, decoupling expressivity gains from rigid positional or probability-encoding constraints. The lag/reverse-offset layout is shown to be essential to preserve AR (autoregressive) extension invariance, ensuring computational consistency and correctness as context grows.

The modeling perspective also gives formal justification to several previously empirical design rules in attention modules, such as the finding that:

• VO-side (readout) rank should be preserved in parameter-constrained settings, as compressing the readout directly restricts the possible update subspace of the residual path.
• Low-rank adapters (e.g., LoRA) and output gating are most effective on the readout side.
• Incorporating explicit temporal (sequence) mixing before/after activation meaningfully expands function class capacity, even at matched parameter budgets.

HyperMLP/HyperGLU Block Design

Building on the dynamic MLP foundation, the authors propose the HyperMLP and its variant HyperGLU. Both heads instantiate their weights via input-conditioned low-rank parameterizations across both feature and sequence axes. The mixing operators $R^{(1)}(x_t)$ and $R^{(2)}(x_t)$ operate in reverse-offset lag order, enabling the canonical AR truncation invariance as theoretically mandated.

For sequence mixing, both diagonal-plus-low-rank (DPLR) forms and optional lightweight convolutions are explored; these components ensure that the memory slot basis can adapt flexibly to input content and context. The GLU-style hidden activation further decouples active-set selection (routing) from slot weighting, improving modulation capability and learning dynamics.

Notably, these blocks can be implemented with only modest $O(T^2r)$ overhead—where $r$ is the low-rank dimension—over standard quadratic attention, and can reuse modern blockwise kernel and memory layouts. However, the new structure cannot directly piggyback on the highly optimized FlashAttention backend.

Empirical Performance: Benchmarks and Insights

The paper presents a thorough empirical study, including both mechanistic synthetic diagnostics (MAD suite) and standard large-scale language modeling tasks (NanoGPT/OpenWebText2 and Fine Web-Edu, with evaluation on the Open LLM Leaderboard and established QA/MC benchmarks).

Key findings are:

• Softmax normalization is not required for strong performance. ReLU-based attention, with appropriate context mixing, matches or exceeds historical softmax baselines.
• Explicit temporal/sequence-space mixing is the dominant driver of observed improvements. Large, often double-digit increases in recall, selective copying, and fuzzy match tasks are observed upon enabling learned sequence mixing.
• The lag layout is critical: Temporal mixing without canonical reverse-offset coordination fails to maintain AR consistency, producing sharp performance drops.
• Two-sided mixing (routing and readout) yields additive gains over one-sided approaches.
• Budget allocation strongly favors preserving readout (VO) rank; compressing the addressing/routing (QK) side is preferable under fixed parameter budgets.
• HyperGLU variant achieves the lowest LM loss and best aggregate benchmark performance under equal-sized parameter settings.

These results are established both in compute-constrained diagnostic settings and at larger, practical model scales, consistently placing HyperMLP/HyperGLU at or near the top of competitive model leaderboards.

Implications, Limitations, and Future Directions

This work challenges the traditional framing of attention as a probabilistic, probability-simplex-normalized lookup, instead reorienting sequence modeling toward context-instantiated dynamic function composition. The theoretical and empirical results jointly reinforce the view that learned temporal and feature mixing in a dynamic-MLP architecture strictly enlarges the expressive envelope of sequence models—enabling superior handling of noisy/fuzzy recall, compression, and complex context-dependent routing.

On the practical side, HyperMLP and HyperGLU deliver consistent capability improvements at fixed compute, and the modular design is directly compatible with Transformer-style residual architectures and standard context-window mechanisms.

However, current implementation optimizations lag behind attention kernels such as FlashAttention, and ablation at frontier model scales remains unexamined due to resource constraints. Further, while the mechanism primarily improves flexible context usage rather than parametric (memorization) capacity, application-dependent concerns such as privacy, bias, and robustness remain as in all powerful sequence models.

Future directions include rigorous scaling experiments at the largest model sizes, hardware-level kernel optimization for fast context mixing, and deeper exploration of the new model's inductive biases in areas such as in-context learning, generalization, and adaptation.

Conclusion

HyperMLP advances an integrated, theoretically grounded rethinking of sequence modeling, casting autoregressive attention as dynamic two-layer MLPs with growing context-dependent hidden representations. The resulting models, HyperMLP and HyperGLU, harness input-conditioned sequence and feature mixing—achieving expressivity and empirical performance surpassing strong softmax-attention baselines at matched compute. This paradigm suggests a promising avenue for further development in efficient and flexible sequence models.