QAS-QTNs: Curriculum Reinforcement Learning-Driven Quantum Architecture Search for Quantum Tensor Networks

Abstract: Quantum Architecture Search (QAS) is an emerging field aimed at automating the design of quantum circuits for optimal performance. This paper introduces a novel QAS framework employing hybrid quantum reinforcement learning with quantum curriculum learning strategies, enabling learning agents to tackle increasingly complex quantum circuit design tasks. We benchmark four state-of-the-art classical reinforcement learning algorithms (A2C, PPO, DDQN, TD3) against their quantum-enhanced counterparts (QA2C, QPPO, QDDQN, QTD3) for optimizing variational quantum circuits (VQCs). Our approach progressively increases circuit depth and gate complexity during training, leveraging parameterized quantum circuits as function approximations. To improve learning efficiency and stability, all algorithms, both classical and quantum, are augmented with Prioritized Experience Replay (PER). Experimental results show that quantum-enhanced RL significantly outperforms classical methods. In a 2-qubit environment, PERQDDQN achieves a success probability of 0.46 with ~3,000 optimal successes, surpassing classical PERDDQN (0.42, ~2,400). In the more complex 3-qubit setting, PERQDDQN and PERQTD3 reach success probabilities of ~0.47, with optimal success counts of ~3,800 and ~3,600, respectively, outperforming their classical counterparts. Additionally, we apply our QAS-QTN approach to a classification problem, where the optimized quantum circuit achieves an accuracy of 90.33\%, outperforming quantum models consisting of random ansatz. This hybrid classical-quantum approach leads to faster convergence and more efficient quantum circuit designs, demonstrating its potential for advancing automated quantum architecture search.