A natural NISQ model of quantum self-attention mechanism

Abstract: The self-attention mechanism (SAM) has demonstrated remarkable success in various applications. However, training SAM on classical computers becomes computationally challenging as the number of trainable parameters grows. Quantum neural networks (QNNs) have been developed as a novel learning model that promises to provide speedup for pattern recognition using near-term Noisy Intermediate-Scale Quantum (NISQ) computers. In this work, we propose a completely natural way of implementing SAM in QNNs, resulting in the quantum self-attention mechanism (QSAM). The fundamental operations of SAM, such as calculating attention scores and producing attention features, are realized by only designing the data encoding and ansatz architecture appropriately. As these are the fundamental components of QNNs, our QSAM can be executed efficiently on near-term NISQ devices. Our QSAM models achieve better performance in terms of both accuracy and circuit complexity on the text categorization task. Moreover, the robustness of QSAM against various typical types of quantum noise is demonstrated, indicating the model's suitability for NISQ devices. The present QSAM will serve as the fundamental building blocks for developing large models of quantum attention neural networks for quantum advantageous applications.