The EnvDesign Model: A Method to Solve the Environment Design Problem

Abstract: Today, several people and organizations rely on cloud platforms. The reliability of cloud platforms depends heavily on the performance of their internal programs (agents). To better prevent regressions in cloud platforms, the design of pre-production testing environments (that test new agents, new hardwares, and other changes) must take into account the diversity of server/node properties (hardware model, virtual machine type, etc.) across the fleet and dynamically emphasize or de-emphasize the prevalence of certain node properties based on current testing priorities. This paper formulates this task as the ``environment design" problem and presents the EnvDesign model, a method that uses graph theory and optimization algorithms to solve the environment design problem. The EnvDesign model was built on context and techniques that apply to combinatorial testing in general, so it can support combinatorial testing in other domains. An earlier version of this paper was peer-reviewed and published internally at Microsoft.