SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

Abstract: Speculative decoding (SD) has emerged as a widely used paradigm to accelerate LLM inference without compromising quality. It works by first employing a compact model to draft multiple tokens efficiently and then using the target LLM to verify them in parallel. While this technique has achieved notable speedups, most existing approaches necessitate either additional parameters or extensive training to construct effective draft models, thereby restricting their applicability across different LLMs and tasks. To address this limitation, we explore a novel plug-and-play SD solution with layer-skipping, which skips intermediate layers of the target LLM as the compact draft model. Our analysis reveals that LLMs exhibit great potential for self-acceleration through layer sparsity and the task-specific nature of this sparsity. Building on these insights, we introduce SWIFT, an on-the-fly self-speculative decoding algorithm that adaptively selects intermediate layers of LLMs to skip during inference. SWIFT does not require auxiliary models or additional training, making it a plug-and-play solution for accelerating LLM inference across diverse input data streams. Our extensive experiments across a wide range of models and downstream tasks demonstrate that SWIFT can achieve over a 1.3x-1.6x speedup while preserving the original distribution of the generated text. We release our code in https://github.com/hemingkx/SWIFT.

• The paper introduces Swift, a novel plug-and-play self-speculative decoding technique that accelerates LLM inference by skipping selected layers without additional training.
• It employs a two-phase process—context-based layer set optimization and confidence-aware inference—to dynamically draft tokens, achieving speedups of up to 1.6x.
• Experiments on LLaMA-2 and CodeLLaMA report token acceptance rates between 98% and 100%, validating the efficiency gains from leveraging inherent model sparsity.

An Analysis of Swift: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

The paper presents "Swift," an innovative approach for enhancing the efficiency of LLM inference through a novel plug-and-play speculative decoding mechanism. This work primarily addresses the inefficiency in autoregressive decoding intrinsic to LLMs, especially as their size increases.

Core Proposal and Methodology

The authors tackle the limitation of existing speculative decoding (SD) methods that necessitate additional parameters or extensive training, which hinders applicability across various models and tasks. They propose a novel approach employing layer-skipping, capitalizing on the inherent sparsity within LLMs. The key innovation, Swift, does not require additional training or auxiliary models, offering a versatile solution for real-time LLM inference acceleration.

Swift operates by dynamically selecting intermediate layers of the LLM to skip during inference, an approach backed by a two-phase inference process:

1. Context-based Layer Set Optimization: This phase involves adaptive optimization of the skipped layer set using the LLM-generated context as a guide, thus achieving efficient token drafting.
2. Confidence-aware Inference Acceleration: Post optimization, the selected layers are leveraged to draft tokens for speculative execution with the final aim of maximizing the acceptance of drafts by the full model.

Experimental Results and Observations

Empirically, Swift demonstrates substantial speedups ranging from 1.3x to 1.6x across various tasks and LLM architectures, such as LLaMA-2 and CodeLLaMA. The token acceptance rate by the full model consistently ranges from 98% to 100% for the LLaMA-2 series, highlighting the effective alignment between the draft and target outputs. Furthermore, the paper reports that Swift's efficiency scales favorably with larger model sizes, suggesting greater sparsity potential.

Theoretical and Practical Implications

Theoretically, Swift offers a unique perspective on self-speculative decoding, revealing that LLMs possess intrinsic sparsity that can be effectively exploited without losing output quality. Practically, the plug-and-play nature of Swift presents an attractive proposition for deploying efficient LLM applications across diverse domains without the extensive overhead typically associated with auxiliary models or retraining.

Future Directions

The paper suggests an intriguing avenue for further research in optimizing the LLM architecture itself, focusing on model sparsity. Future work could extend Swift's methodology to investigate its applicability to even larger models and the potential integration with other speculative decoding paradigms, such as Jacobi-based methods, for enhanced performance.

In conclusion, Swift presents a compelling advancement in the field of LLM inference acceleration, demonstrating both theoretical insight and practical efficiency. Its contributions lay the groundwork for further investigation into the underexplored field of adaptive plug-and-play solutions for LLMs.