On the Computational Hardness of Transformers

Abstract: The transformer has revolutionized modern AI across language, vision, and beyond. It consists of $L$ layers, each running $H$ attention heads in parallel and feeding the combined output to the subsequent layer. In attention, the input consists of $N$ tokens, each a vector of dimension $m$. The attention mechanism involves multiplying three $N \times m$ matrices, applying softmax to an intermediate product. Several recent works have advanced our understanding of the complexity of attention. Known algorithms for transformers compute each attention head independently. This raises a fundamental question that has recurred throughout TCS under the guise of ``direct sum'' problems: can multiple instances of the same problem be solved more efficiently than solving each instance separately? Many answers to this question, both positive and negative, have arisen in fields spanning communication complexity and algorithm design. Thus, we ask whether transformers can be computed more efficiently than $LH$ independent evaluations of attention. In this paper, we resolve this question in the negative, and give the first non-trivial computational lower bounds for multi-head multi-layer transformers. In the small embedding regime ($m = N^{o(1)}$), computing $LH$ attention heads separately takes $LHN^{2 + o(1)}$ time. We establish that this is essentially optimal under SETH. In the large embedding regime ($m = N$), one can compute $LH$ attention heads separately using $LHN^{ω+ o(1)}$ arithmetic operations (plus exponents), where $ω$ is the matrix multiplication exponent. We establish that this is optimal, by showing that $LHN^{ω- o(1)}$ arithmetic operations are necessary when $ω> 2$. Our lower bound in the large embedding regime relies on a novel application of the Baur-Strassen theorem, a powerful algorithmic tool underpinning the famous backpropagation algorithm.

• The paper shows that under SETH and k-OV assumptions, computing all attention heads independently is essentially optimal.
• It leverages reductions from orthogonal vectors and the Baur-Strassen theorem to establish tight lower bounds in both small and large embedding regimes.
• The findings imply that accelerating transformer computations requires approximations, hardware innovations, or architecture changes rather than new exact algorithms.

On the Computational Hardness of Transformers

Problem Overview and Motivation

The transformer architecture underpins state-of-the-art systems in NLP and CV, driven by multi-layer, multi-head attention mechanisms. Despite the empirical success, fundamental algorithmic questions around the computational efficiency of transformers remain unresolved. Specifically, can multiple attention heads and layers be computed substantially faster than their naive independent evaluation? This question aligns with the classic direct sum problem in theoretical computer science—whether multiple instances of a problem can be solved more efficiently in parallel than in sequence.

The paper "On the Computational Hardness of Transformers" (2603.11332) provides the first non-trivial lower bounds for the computational complexity of transformers. It establishes, under SETH and related fine-grained complexity assumptions, that the standard method of independently computing all attention heads in all layers is essentially optimal. Notably, the work precisely matches upper and lower bounds in both the small and large embedding regimes, using tools from fine-grained complexity and circuit complexity.

Main Results and Contributions

1. Lower Bounds—Small Embedding Regime:

When the embedding dimension $m$ is subpolynomial in the sequence length $N$ (specifically $m = N^{o(1)}$), the naive algorithm for an $L$-layer, $H$-head transformer computes all attention heads independently in $O(LHN^{2+o(1)})$ time. Assuming the Strong Exponential Time Hypothesis (SETH), the orthogonal vectors hypothesis (specifically for $k=3$), and reductions from unbalanced $k$-OV, the authors show that any algorithm (not even necessarily using the transformer structure) must use at least $LHN^{2-o(1)}$ time. Thus, amortized or parallel computation yields no asymptotic speedup over the naive baseline.

2. Lower Bounds—Large Embedding Regime:

When the embedding dimension $m = N$, multi-head attention can leverage fast matrix multiplication to improve upon the naive cubic time per head. The best-known algorithm achieves $LHN^{\omega+o(1)}$, where $\omega$ is the matrix multiplication exponent. The paper proves this is also essentially optimal in the extended arithmetic circuit model—even when exponentiation and logarithm gates are available—using a novel reduction via the Baur-Strassen theorem. The reduction shows that an extended arithmetic circuit computing such a transformer must be large enough to compute $LH$ independent instances of matrix multiplication, incurring $LHN^{\omega-o(1)}$ gate complexity whenever $\omega > 2$.

3. Extended Circuit Lower Bounds:

A key technical contribution is the extension of classical arithmetic circuit lower bounds to circuits augmented with non-algebraic gates (such as $\exp$ and $\ln$), motivated by the structure of the attention operation. The work proves that these gates offer no advantage for computing low-degree functions (such as matrix multiplication outputs and derivatives), by analyzing their formal power series and leveraging extended versions of Strassen’s results.

4. Tightness and Matching Upper Bounds:

For both small and large embedding regimes, the presented lower bounds closely match existing upper bounds, modulo small (polylogarithmic) factors. Thus, improvements—conditional on the considered complexity assumptions—are excluded even with sophisticated circuit types and access to fast algebraic operations.

Notable Technical and Conceptual Innovations

• Reduction from Orthogonal Vectors (OV) and $k$-OV: The lower bounds for the small embedding regime exploit reductions from unbalanced $k$-OV instances, showing that solving all attention heads "in parallel" cannot break the quadratic (in $N$) time barrier without refuting SETH or the $k$-OV hypothesis.
• Use of the Baur-Strassen Theorem: For the large embedding regime, the authors generalize Baur-Strassen to the extended arithmetic circuit model to extract enough information (partial derivatives) from a “compressed” computation. They show that, to produce all outputs for $LH$ matrix multiplications, the circuit must have enough gates to match the trivial approach, even allowing rich gatesets.
• Treatment of Practical MLPs and Activation Functions: The lower bounds are robust to the inclusion of standard MLP layers (including GLUs, ReLU, and sigmoid), provided the computation remains of low circuit complexity per layer, further emphasizing the universality of the hardness results.

Implications

Theoretical Implications:

The results indicate that the quadratic (small $m$) or $N^\omega$ (large $m$) scaling of transformer computation is not an artifact of implementation, but a fundamental computational barrier—conditional on SETH and matrix multiplication lower bounds. Furthermore, this places transformer computation solidly among the growing class of problems for which fine-grained complexity precludes radical algorithmic improvements unless major complexity-theoretic conjectures are refuted.

Practical Implications:

Efforts to accelerate transformer inference or training are compelled to rely on either hardware specialization, approximation, or architectural changes (e.g., sparse or low-rank attention), as there are no unconditional algorithmic savings to be had under standard complexity assumptions. The work directly rules out amortized improvements for large batch inference or speculative “multi-head fusion” optimizations presently pursued in systems and compiler communities.

Directions for Future Work:

The paper closes the door on algorithmic advances for exact transformer computation under the model and assumptions considered but leaves open alternative directions:

• Randomized/Approximate Computation: Can efficient approximations circumvent these barriers without incurring significant accuracy degradation?
• Other Computational Models: Extending unconditional hardness to the Word-RAM model or to practical hardware-focused models is left for future investigation.
• Distributional and Structural Assumptions: Might real-world data distributions admit faster (even average-case) transformer algorithms? Structural restrictions on weights or inputs may open the door for more efficient computation.

Conclusion

"On the Computational Hardness of Transformers" rigorously characterizes the computational limits of exact transformer evaluation, grounding the anecdotal inefficiency of large-scale attention computation in fine-grained complexity assumptions and non-trivial lower bounds for extended arithmetic circuits. The paper closes the possibility of subquadratic or sub-$N^\omega$ evaluation algorithms for multi-layer, multi-head transformers under accepted complexity hypotheses, shaping future work toward architectural, hardware, or approximate solutions.