BDD e2e test

[eytan-avisror]

governor packages should have a end to end functional test.
we can have a travis cronjob that runs nightly and run this test

test should:

• setup eks cluster
• test pod-reaper:
  • create a pod with infinite sleep, terminate it - it is now stuck, let reaper kill it
• test node-reaper:
  • schedule a pod on a node with hostNetwork and run ip link set dev eth0 down, node will become NotReady, let reaper kill it
  • Test age-reap by setting a low threshold
  • Test flappy node reap by faking events or restarting kubelet on the node

[pratyushprakash]

Can I take this up?

[eytan-avisror]

@pratyushprakash Sure! let's have a discussion on the design so that we are aligned.

Maybe a good start is a separate package that takes in a kubeconfig or uses active cluster context if not provided to run tests. this can live under /test-bdd/main_test.go and invoke the tests in individual files e.g. /test-bdd/podreaper.go

some basic example exists here https://github.com/keikoproj/instance-manager/tree/master/test-bdd

An easy start is maybe doing a few tests scenarios for pod-reaper which should be much easier than testing node-reaper.

Suggest we use Gingko (https://github.com/onsi/ginkgo) but open for other frameworks if you have any preference.

Scenarios we should look at for pod reaper:

• A pod which is stuck in Terminating state is reapable after N2 minutes (depending on --reap-after flag)
• A pod which is stuck in Terminating state is not reapable after N1 minutes (depending on --reap-after flag)
• If dry-run is enabled, reapable pods are not reaped
• (--soft-reap=true) if containers are still running, pods are not reaped

How to reproduce a stuck pod for testing

• Run some pod that is doing some infinite 'sleep' and issue a terminate command to it
  It will then go into terminating state and stay in that state (with container running) until sleep is done. This means as long as --soft-reap is off, it will become reapable.
  We can test that if --soft-reap is on, pod does not get deleted (testing the inverse of this will be difficult).

Once we have the code, we can add separate issue for making sure it runs nightly in travis-ci.

The above should be an easy start at this, WDYT @pratyushprakash ?