NDBench: Benchmarking Microservices at Scale

Abstract: Software vendors often report performance numbers for the sweet spot or running on specialized hardware with specific workload parameters and without realistic failures. Accurate benchmarks at the persistence layer are crucial, as failures may cause unrecoverable errors such as data loss, inconsistency or corruption. To accurately evaluate data stores and other microservices at Netflix, we developed Netflix Data Benchmark (NDBench), a Cloud benchmark tool. It can be deployed in a loosely-coupled fashion with the ability to dynamically change the benchmark parameters at runtime so we can rapidly iterate on different tests and failure modes. NDBench offers pluggable patterns and loads, support for pluggable client APIs, and was designed to run continually. This design enabled us to test long-running maintenance jobs that may affect the performance, test numerous different systems under adverse conditions, and uncover long-term issues like memory leaks or heap pressure.