Continuous evaluation of the performance of cloud infrastructure for scientific applications

Abstract: Cloud computing recently developed into a viable alternative to on-premises systems for executing high-performance computing (HPC) applications. With the emergence of new vendors and hardware options, there is now a growing need to continuously evaluate the performance of the infrastructure with respect to the most commonly-used simulation workflows. We present an online ecosystem and the corresponding tools aimed at providing a collaborative and repeatable way to assess the performance of the underlying hardware for multiple real-world application-specific benchmark cases. The ecosystem allows for the benchmark results to be stored and shared online in a centrally accessible database in order to facilitate their comparison, traceability, and curation. We include the current up-to-date example results for multiple cloud vendors and explain how to contribute new results and benchmark cases.