Adaptive Shot Allocation for Recursive QAOA via Reinforcement Learning

Abstract: Recursive QAOA (RQAOA) solves combinatorial optimization problems by using shallow quantum circuits to estimate pairwise correlations and recursively eliminate variables until a classical solver can handle the residual instance. Each elimination step requires measurement shots, and the total shot cost grows with the number of recursive stages. On near-term quantum devices, increasing shot counts can translate directly into greater exposure to hardware-level noise sources such as readout errors and decoherence, making shot-efficient execution not merely a cost-reduction measure but a factor with direct implications for solution reliability. While shot reduction has been studied broadly across NISQ algorithms, step-wise measurement control inside the recursive loop of RQAOA has received little attention. We formulate this step-wise allocation as a sequential decision problem and propose two strategies for depth-1 RQAOA on weighted Max-Cut instances. A hand-crafted heuristic assigns shots based on local indicators of step difficulty, and a tabular Double Q-learning agent learns a residual policy that adjusts this baseline under a Lagrangian-constrained objective. Both methods are evaluated under a fixed-cap fairness protocol that equalizes the per-step budget across all strategies, and the elimination rule itself is kept unchanged so that the contribution of adaptive measurement control can be isolated. On a diverse set of weighted graph instances spanning a range of sizes and structures, the heuristic reduces total shots by approximately 23% relative to uniform allocation, and the RL policy achieves a 36% reduction with a lower effective shots per success ratio than both baselines. The improvement persists on problem sizes not seen during training, suggesting that reinforcement learning can discover efficient, instance-adaptive measurement strategies in recursive quantum optimization.

• The paper demonstrates that a reinforcement learning approach can reduce shot usage by 36.1% compared to uniform allocation in RQAOA.
• The methodology combines a heuristic baseline with a residual Double Q-learning agent to dynamically adjust shot allocation based on local step difficulty.
• Quantitative results show that the RL policy maintains success rates of 91–95% and generalizes effectively across different problem sizes.

Adaptive Shot Allocation for Recursive QAOA via Reinforcement Learning

Introduction and Motivation

Recursive Quantum Approximate Optimization Algorithm (RQAOA) is an advanced hybrid quantum-classical approach for solving combinatorial optimization problems, notably weighted Max-Cut. RQAOA recursively applies low-depth QAOA to reduce problem size via high-confidence pairwise correlation measurements, ultimately enabling a classical solver to process the residual instance efficiently. While the recursive structure circumvents certain locality and trainability bottlenecks of low-depth QAOA, each contraction stage introduces a shot-expense bottleneck: repeated circuit executions accumulate significant shot cost, magnifying the impact of hardware noise and decoherence on NISQ devices. Constructing efficient step-wise shot allocation policies that adapt to local step difficulty is therefore vital, yet little attention has been paid to recursion-level measurement control in RQAOA.

Background: Max-Cut and RQAOA Architecture

Max-Cut is framed in weighted Ising form and solved via depth-1 QAOA. At each elimination step, the pairwise correlation matrix is estimated using finite measurement shots. The edge with maximum absolute correlation is selected to contract, and the graph is recursively reduced. Typically, uniform shot allocation is employed for all steps. However, the difficulty of elimination steps varies depending on correlation separation and endpoint conflict, motivating state-dependent allocation strategies.

Proposed Shot-Control Framework

The step-wise shot allocation in RQAOA is formulated as a finite-horizon Markov Decision Process (MDP). The allocation controller operates in two modes:

• Heuristic Baseline: Domain-informed rule based on local indicators of step difficulty (z-gap, conflict ratio, edge distance) assigns allocation fractions from a predefined candidate set.
• Residual Double Q-learning Agent: Tabular reinforcement learning agent refines the Heuristic output by learning a residual offset. The agent’s reward penalizes shot usage subject to an adaptive Lagrangian reliability constraint. State descriptors are efficiently extracted via small probe measurements prior to main shot allocation.

The complete shot-control loop integrates into the RQAOA pipeline without altering the elimination rule. This modularity enables plug-in replacement for uniform allocation, facilitating practical deployment.

Figure 1: Overview of the proposed shot-control framework for depth-1 RQAOA, detailing the recursive reduction and RL-guided step-wise allocation.

Experimental Design

Training and evaluation are conducted entirely on noiseless Qiskit Aer simulation. Two base policies are trained on hard $d$-regular weighted Max-Cut graph instances, selected for high operational difficulty. Cap calibration is performed per-instance using uniform allocation to ensure fairness. The benchmark covers 104 unique graph instances and over 160 policy-instance pairs, including cross-size transfer evaluation. Three allocation strategies (Uniform, Heuristic, RL) are compared over 60 independent runs per pair.

Quantitative Results

Shot Reduction and Effective Cost

On the held-out operational subset (130 pairs), the RL policy achieves 36.1% shot reduction relative to Uniform, compared to 23.0% for the Heuristic. RL lowers the effective shots per success (ESP ratio) to 0.676 (relative to Uniform), outperforming both baselines in both same-size and unseen-size regimes.

Figure 2: Shot reduction (\%) versus problem size $n$, demonstrating RL’s stable 36\% reduction across the operational benchmark.

Figure 3: Normalized shot usage ($$\widetilde{S}_{\text{method}}/\widetilde{S}_{\text{Uniform}}$$) versus success rate, visualizing the RL method’s shift toward lower shot consumption at competitive reliability.

Figure 4: Mean ESP ratio by problem size $n$, confirming consistently lower effective measurement cost per successful solution for RL across all evaluated sizes.

Robustness and Reliability

Adaptive Lagrangian constraint maintains reliability: RL controller achieves average success rates near 0.91–0.95 on the operational subset, and outperforms unconstrained variants by a substantial margin at strict reliability floors. Metric sensitivity analysis confirms that RL’s shot reduction is robust across mean, median, tail-sensitive (P90), and restart-cost metrics. Penalty regime sensitivity tests show negligible impact on the learned allocation policy’s main performance indicators.

Training Dynamics

Training diagnostics demonstrate policy convergence: under progressive tightening of the Lagrangian penalty, RL maintains the target success rate while sustaining aggressive shot reduction.

Figure 5: Training diagnostics for the $b_0$ policy ($n{=}14$), showing success rate stabilization and adaptive Lagrangian multiplier escalation.

Per-Step Allocation Behavior

The residual RL controller exhibits higher selectivity: it invests minimum allocation on easy steps matching the baseline, while substantially increasing allocation to hard steps. Direct-action controllers lack this discrimination, reducing performance.

Figure 6: Per-step allocation pattern by step difficulty, illustrating selective shot investment by the residual controller.

Discussion and Implications

The results isolate and quantify the contribution of adaptive measurement control within the recursive structure of RQAOA. The Heuristic baseline achieves substantial resource savings, but RL consistently refines this prior, improving efficiency without compromising reliability. Notably, RL’s advantage increases with problem size and recursion depth, aligning with practical NISQ quantum optimization interests. The modular architecture enables straightforward deployment as a plug-in component in existing RQAOA pipelines. Transferability across problem sizes validates that the state representation encodes local step difficulty rather than memorizing graph-size heuristics.

From a broader perspective, recursion-level shot allocation complements circuit-level and parameter-level optimization, filling a gap in measurement-frugal quantum algorithm design for resource-constrained devices. Limitations include restriction to noiseless simulation and depth-1 QAOA; extensions to deeper circuits, hardware validation, and function-approximation-based RL controllers are logical directions.

Conclusion

This work formalizes step-wise shot allocation in depth-1 RQAOA as a sequential decision problem and demonstrates that reinforcement learning, refined via residual Double Q-learning, achieves substantial measurement cost reduction—36\%—over uniform allocation with competitive reliability. The learned controller generalizes to unseen-size instances, confirming the effectiveness of the local state encoding. Experimental analyses underscore the stability, interpretability, and practical deployability of the approach. Future research should address higher QAOA depths, quantum hardware integration, robust multi-instance policy learning, and diverse graph classes, expanding the scope and impact of adaptive recursion-level measurement allocation in variational quantum optimization.