Monitoring Cascading Changes of Resources in the Kubernetes Control Plane

Abstract: Kubernetes is a container management system that has many automated functionalities. Those functionalities are managed by configuring objects and resources in the control plane. Since most objects change their state depending on other objects' states, a change propagates to other objects in a chain. As cluster availability is influenced by the time required for these cascading changes, it is essential to make the propagations measurable and shed light on the behavior of the Kubernetes control plane. However, it is not easy because each object constantly monitors other objects' status and acts autonomously in response to their changes to play its role. In this paper, we propose a measurement system that outputs objects' change logs published from the API server in the control plane and assists in analyzing the time of cascading changes between objects by utilizing the relationships among resources. With a practical change scenario, our system is confirmed that it can measure change propagation times within a cascading change. Also, measurements on the system itself showed it has a small CPU and memory footprint.