Benchmarking a quantum annealing processor with the time-to-target metric

Abstract: In the evaluation of quantum annealers, metrics based on ground state success rates have two major drawbacks. First, evaluation requires computation time for both quantum and classical processors that grows exponentially with problem size. This makes evaluation itself computationally prohibitive. Second, results are heavily dependent on the effects of analog noise on the quantum processors, which is an engineering issue that complicates the study of the underlying quantum annealing algorithm. We introduce a novel "time-to-target" metric which avoids these two issues by challenging software solvers to match the results obtained by a quantum annealer in a short amount of time. We evaluate D-Wave's latest quantum annealer, the D-Wave 2X system, on an array of problem classes and find that it performs well on several input classes relative to state of the art software solvers running single-threaded on a CPU.

• The paper introduces the time-to-target (TTT) metric to overcome limitations of traditional quantum annealing benchmarks.
• It experimentally compares the D-Wave 2X against classical solvers, revealing 2x to 600x faster performance in specific problem instances.
• The study highlights TTT as a scalable, practical approach for real-world quantum benchmarking and future multi-thread/GPU evaluations.

Analysis of the "Time-to-Target" Metric for Benchmarking Quantum Annealers

The paper "Benchmarking a quantum annealing processor with the time-to-target metric" introduces a novel metric for evaluating the performance of quantum annealers, particularly the D-Wave 2X system. This "time-to-target" (TTT) metric provides an alternative to traditional metrics based on ground state success rates, addressing two critical issues associated with prior methodologies: the exponential growth of computational time required for evaluation as problem sizes increase, and the sensitivity of results to analog noise in quantum processors.

Core Contributions

The authors present several key contributions through this paper:

1. Novel Metric Introduction: By implementing the TTT metric, the authors provide a new paradigm that forces classical algorithms to match the performance of a quantum annealer within specific time constraints, effectively circumventing issues related to analog noise and prohibitive runtime.
2. Experimental Evaluation: The paper meticulously evaluates D-Wave's 2X quantum annealer against state-of-the-art classical software solvers on various problem instances characterized by different levels of complexity. This comparative study presents the advantage of quantum processors in situations where classic solvers struggle under limited time constraints.
3. Comparison with Existing Methods: Previous benchmarks such as the samples-to-solution (STS) and time-to-solution (TTS) have been shown to struggle with large problem sizes due to computational infeasibility. The TTT metric is posited as a solution to this problem, allowing for performance comparison without needing to reach a ground state, which is computationally demanding for large problems.

Numerical Outcomes and Claims

Through structured experimentation, the authors highlight their core findings:

• The D-Wave 2X demonstrated significant computational advantages over classical solvers, with TTT times being 2x to 600x faster depending on whether input/output (I/O) costs were considered.
• For certain problem classes, primarily those leveraging the strengths of quantum annealing on specific graph structures like Chimera, the D-Wave 2X's performance was notably superior.

Implications and Future Directions

The practical implications of the presented research are substantial for the quantum computing domain. The TTT metric enables a fairer, more realistic assessment of quantum annealers in scenarios that closely mimic real-world application where system precision and robustness are essential. The introduced methodology paves the way for future benchmarks to incorporate quantum methods as viable alternatives to classical solutions across various disciplines.

On a theoretical level, the usage of alternative metrics like TTT can reshape foundational understandings of quantum annealer capabilities, particularly their potential applications in optimization problems traditionally addressed using classical computing resources.

Looking forward, the field should explore extending the TTT concept to multi-core and multi-threaded classical solvers, including GPU-based solutions, to further evaluate quantum annealer performance. This inquiry could reveal broader areas where quantum supremacy might be achievable, moving beyond single-threaded comparisons to encompass parallel processing capabilities.

This paper represents a meaningful advance in quantum computing performance evaluation, significantly contributing to understanding the practical applications and limitations of current quantum annealing technologies. Such insights are crucial as the industry navigates the integration of quantum computing solutions.