EfficientASR: Compressed Transformer ASR

Updated 7 May 2026

EfficientASR is a framework that streamlines Transformer-based ASR by leveraging shared multi-head attention and chunk-level FFN to lower computational costs.
It achieves up to 36% parameter reduction and halves FLOPs, enabling efficient training and deployment even for long utterances and resource-constrained devices.
The modular design supports hardware-aware optimizations and faster decoding while maintaining or improving recognition accuracy.

EfficientASR encompasses a body of algorithmic, architectural, and practical methods for compressing, accelerating, and scaling automatic speech recognition neural networks—most prominently Transformer-based and encoder-decoder ASR models—while maintaining or even improving recognition accuracy. The term refers not to a single architecture but to a research space emphasizing structural network modifications (attention/computation sharing, chunked/layer-wise compression), efficient training and scaling strategies, discrete or low-rank audio representations, and hardware-aware optimizations. This entry reviews the EfficientASR paradigm with a focus on the canonical network compression technique of "Shared Residual Multi-Head Attention and Chunk-Level FFN," contextualizing it among other major efficient ASR strategies.

The "EfficientASR" model (Wang et al., 2024) is a lightweight Transformer-based end-to-end ASR system specifically engineered for computational and memory efficiency. Its structural innovations target both major computational bottlenecks of the Transformer:

Shared Residual Multi-Head Attention (SRMHA):

SRMHA targets redundant recomputation across layers within the multi-head attention (MHA) mechanism. In standard Transformers, each encoder layer recomputes the full $QK^T$ attention map per head, incurring $O(T^2)$ costs per layer (for sequence length $T$ ). SRMHA introduces two operational modes—"updated" (recomputing $QK^T$ and fusing with the previous map via a residual) and "shared" (simply reusing the prior attention map and recalculating only $V$ -related projections). This halves both the parameter count and the attention-related FLOPs in practice. Additionally, a sliding-window-with-deformability (SWD) mask restricts attention to local windows, reducing the quadratic $T^2$ term to $O(Tw)$ for window size $w \ll T$ .

Chunk-Level Feedforward Network (CFFN):

The canonical Transformer feedforward network (FFN) is parameter-intensive, typically scaling as $2d \cdot d_{ff}$ per layer (with $d_{ff} \gg d$ ). CFFN uniformly splits the hidden activations along the embedding axis into $O(T^2)$ 0 chunks. Each chunk is processed by a much smaller "mini-FFN"—parameter count and FLOPs shrink by a factor $O(T^2)$ 1 relative to the standard FFN. CFFN thus systematically reduces network size and computation, and the optimum chunking factor trades parameter savings against potential accuracy loss depending on data regime.

Integration:

EfficientASR assembles these modifications into an ASR pipeline where input features traverse through convolutional downsampling, positional encoding, $O(T^2)$ 2 stacked [SRMHA $O(T^2)$ 3 SWD $O(T^2)$ 4 CFFN] blocks, and a cross-attentional decoder (which may also use SRMHA). Training can use a joint CTC+cross-entropy objective; inference is compatible with beam search and external language modeling.

2. Quantitative Gains: Parameter, FLOP, and Memory Reduction

Experimental results on Mandarin Aishell-1 and HKUST datasets, comparing EfficientASR to a standard Transformer baseline ( $O(T^2)$ 5, $O(T^2)$ 6, 6 encoder/6 decoder layers), provide:

Model	Params (M)	Test CER (%) (Aishell-1)	Test CER (%) (HKUST)
Transformer	30.35	5.9	— (21.7 dev)
EfficientASR (full)	19.33	5.6	— (22.0 dev)
EfficientASR + LM	—	5.3	21.3

These results reflect a 36% parameter reduction (19.33M vs. 30.35M), accompanied by a CER improvement (–0.3% Aishell-1 test, –0.2% HKUST dev). FLOPs per encoder-decoder stack are halved due to SRMHA and CFFN. Memory usage on GPU decreases by up to 30% for utterances exceeding 10 seconds.

Module-level ablations show that SRMHA and CFFN individually provide approximately 50% parameter savings in their respective submodules. Deep empirical sweeps confirm that accuracy remains robust for moderate chunk sizes ( $O(T^2)$ 7); extremely aggressive chunking ( $O(T^2)$ 8) may cause degradation on low-resource tasks.

3. Training, Evaluation, and Practical Deployment

EfficientASR is implemented in ESPnet v2. Standard feature extraction—80-dim log-Mel filterbanks with SpecAugment and convolutional downsampling—is employed. CTC weighting is set to 0.3 (train), increased to 0.6 (inference). Adam optimization, dropout, and label smoothing are used. Beam widths of 10 (search) and 16-layer Transformer LLMs (LM) are supported.

Deployment advantages center on significant reductions in parameter footprint and memory allocation, which directly impact batch size, sequence length handling, and device portability. For long-form transcriptions or edge deployments, these savings translate into persistent cost, latency, and memory advantages.

4. Comparison With Other EfficientASR Paradigms

EfficientASR's compression strategy is orthogonal but complementary to several other efficiency-focused research lines:

Low-rank compression (LiteASR): Uses PCA- and SVD-driven factorization for all linear transforms and internal activation spaces, reducing encoder model size by up to 50% in Whisper-like models without WER loss (Kamahori et al., 27 Feb 2025).
Discrete representation pipelines (Codec-ASR, SSL+kmeans): Replace float-valued frames with quantized or codebook-embedded speech features, reducing both storage and compute with minor accuracy loss (Dhawan et al., 2024, Chang et al., 2023).
Decoding acceleration (diffusion/NAR/block-based/integrated tripartite): Employs non-autoregressive decoders (e.g., blockwise AMD, masked diffusion) to realize parallel or block-parallel token emission, dramatically reducing inference latency at matched or better WER (Wang et al., 12 Nov 2025, Wang et al., 2024, Yen et al., 21 Feb 2026, Tian et al., 25 Jan 2026).
Runtime adaptation/hardware co-design: Dynamic quantization (guided by beam search confidence), CGLA/ASIC mapping, and related hardware-aware optimizations reduce runtime power and total energy consumption in embedded scenarios (Ando et al., 4 Nov 2025, Pinto et al., 2021).

EfficientASR's approach is distinctive in that it applies lightweight functional modularity to the network's core attention and FFN, rather than relying on external compression or runtime adaptation alone.

5. Limitations, Sensitivities, and Open Questions

While EfficientASR provides robust gains on several Mandarin corpora, several caveats and pending research questions are noted (Wang et al., 2024):

Sensitivity to chunking (CFFN): Extremely small chunk sizes benefit parameter count but may harm recognition on low-resource or highly variable datasets.
Scheduling of SRMHA modes: The optimal ratio and placement of "shared" vs. "updated" attention layers is likely data-dependent; further theoretical and practical analysis is needed.
Streaming/Transducer generality: While designed for Transformer encoder-decoders with cross-attention, it remains to be determined whether SRMHA/CFFN can be directly integrated with streaming or RNN-T architectures.
Language/Domain transfer: Most results are in Mandarin; careful benchmarking is needed for diverse language and speaking conditions.

6. Broader Context and Relationship to Efficient Neural Modeling

EfficientASR is part of a general pattern in modern ASR research: leveraging domain knowledge—about the structure of attention, redundancy, and feature formation—instead of blind parameter scaling. The empirical result is a ~36% overall parameter and ~50% computation reduction with no loss of (and sometimes improvement in) character or word error rates. Similar philosophies drive recent work in LLM-based scaling (Mu et al., 6 Aug 2025), integrated non-AR decoding (Wang et al., 12 Nov 2025), and hardware-aware methods (Ando et al., 4 Nov 2025).

The EfficientASR system architecture and its core module designs are applicable beyond ASR, offering generalizable templates for sequence modeling tasks where attention redundancy and FFN bloat are encountered.

7. Summary Table: EfficientASR Module Impact

Module	Param Change	FLOP Change	CER/WER Impact
SRMHA	–50% (Att)	–50% (Att)	Maintained/slight gain
CFFN (n=2)	–50% (FFN)	–50% (FFN)	Maintained/slight gain
SRMHA+CFFN	–36% (Net)	–50% (Total)	–0.3%/–0.2%