Multi-controller Support for Running Sieve

[kosyd]

⭐ Following up from NA KubeCon 2021 ⭐
Per my discussion with Lalith Suresh and Xudong Sun

Is there a way to test race conditions between multiple controllers running in the same Kubernetes cluster simultaneously with Sieve? Testing each controller independently may work as an alternative approach but it can become impractical given the number of controllers that may be running in a given Kubernetes cluster and the desired number of test workloads to be tested.

For example, testing the HPA controller, VPA controller, and Cluster Autoscaling controller along with a custom controller/operator requires testing each component separately for each test workload you'd like to check.

[marshtompsxd]

@kosyd Thank you so much for writing the issue! And sorry for the late reply.

Although so far we have been testing each custom controller separately, there is actually nothing preventing us from testing multiple controllers at the same time. One can set up a test environment with multiple different controllers running and Sieve will focus on injecting faults into one controller.

The real difficulty is the combinatorial explosion issue if we want to inject fault to multiple controllers. Let's say the number of fault injections Sieve wants to conduct for one controller is X1 (which is usually around 50 ~ 200). If we want to try all the possible combinations of fault injections for n controllers, the total number of fault injections we need to conduct will be X1*X2*...*Xn, which can be a huge number. We are trying to figure out how to make it more practical.

The data race issue between multiple controllers you mentioned is a great example that we can start from. If we focus on manipulating the order between read/write (on the same resource) issued by multiple controllers, testing multiple controllers should become much more practical. We will put this testing-data-race-cross-controllers task into our TO-DO list and will let you know if we have any progress on this.