Training Hybrid Deep Quantum Neural Network for Efficient Reinforcement Learning

Abstract: Quantum circuits embed data in a Hilbert space whose dimensionality grows exponentially with the number of qubits, allowing even shallow parameterised quantum circuits (PQCs) to represent highly-correlated probability distributions that are costly for classical networks to capture. Reinforcement-learning (RL) agents, which must reason over long-horizon, continuous-control tasks, stand to benefit from this expressive quantum feature space, but only if the quantum layers can be trained jointly with the surrounding deep-neural components. Current gradient-estimation techniques (e.g., parameter-shift rule) make such hybrid training impractical for realistic RL workloads, because every gradient step requires a prohibitive number of circuit evaluations and thus erodes the potential quantum advantage. We introduce qtDNN, a tangential surrogate that locally approximates a PQC with a small differentiable network trained on-the-fly from the same minibatch. Embedding qtDNN inside the computation graph yields scalable batch gradients while keeping the original quantum layer for inference. Building on qtDNN we design hDQNN-TD3, a hybrid deep quantum neural network for continuous-control reinforcement learning based on the TD3 architecture. On the high-dimensional Humanoid-v4 benchmark, our agent reaches a test return that surpasses classical TD3, SAC and PPO baselines trained with identical compute. To our knowledge this is the first PQC-enhanced policy that matches or exceeds state-of-the-art classical performance on Humanoid. qtDNN has the potential to reduce quantum-hardware calls significantly and is designed to be compatible with today's NISQ devices. The method opens a path toward applying hybrid quantum models to large-scale RL and other gradient-intensive machine-learning tasks.