Trying to Knock Down ETCD with KWOK

[Sharpz7]

See some relevant work:

• Handle Extremely High Throughput by holding back requests to etcd until the throughput decreases. etcd-io/etcd#16837 trying to explain what we observed using and creating https://github.com/armadaproject/armada to the etcd maintainers.
• https://github.com/Sharpz7/kwok-etcd my starter code for trying to knock down etcd.

What kind of pod churn am I going to need? Do I need to have my pods just have more data in them (i.e filled with metadata?)

I will hopefully add to this thread overtime with my findings.

[wzshiming]

apiserver has a QPS limit, a single kwok-controller should not be able to easily bring down etcd, I think may need to try disabling the QPS limit in an existing cluster and start a few more kwok-controller to put enough pressure on etcd.