Classical optimization algorithms for diagonalizing quantum Hamiltonians

Abstract: Diagonalizing a Hamiltonian, which is essential for simulating its long-time dynamics, is a key primitive in quantum computing and has been proven to yield a quantum advantage for several specific families of Hamiltonians. Yet, despite its importance, only a handful of diagonalization algorithms exist, and correspondingly few families of fast-forwardable Hamiltonians have been identified. This paper introduces classical optimization algorithms for Hamiltonian diagonalization by formulating a cost function that penalizes off-diagonal terms and enforces unitarity via an orthogonality constraint, both expressed in the Pauli operator basis. We pinpoint a class of Hamiltonians that highlights severe drawbacks of existing methods, including exponential per-iteration cost, exponential circuit depth, or convergence to spurious optima. Our approach overcomes these shortcomings, achieving polynomial-time efficiency while provably avoiding suboptimal points. As a result, we broaden the known realm of fast-forwardable systems, showing that quantum-diagonalizable Hamiltonians extend to cases generated by exponentially large Lie algebras. On the practical side, we also present a randomized-coordinate variant that achieves a more efficient per-iteration cost than the deterministic counterpart. We demonstrate the effectiveness of these algorithms through explicit examples and numerical experiments.