kubo-disaster-recovery-acceptance-tests (k-DRATs)

Tests a given CFCR K8s cluster can be backed up and restored using bbr.

The acceptance test suite provides hooks around bbr director backup and bbr director restore.

Destructive: do not run in parallel with other tests that create/validate K8s objects

IMPORTANT: This suite backs up and restores the universe of K8s. If it is run in parallel with other tests that create and then validate objects, there can be a race condition wherein those objects are removed before their subsequent restoration, causing unreliable, "flake", false-positive, failed results.

Dependencies

1. Install Golang
2. Install ginkgo CLI
3. Install kubectl CLI
4. Download bbr CLI and add it the PATH

Running k-DRATs in your pipelines

We encourage you to use our set-kubconfig and run-k-drats CI tasks to run k-DRATS in your Concourse pipeline.

Please refer to our k-drats pipeline definition for a working example.

Running k-DRATs locally

1. Spin up a CFCR deployment
   • kubo-deployment is supported. Make sure you apply the enable-bbr opsfile at deploy time to ensure the backup and restore scripts are enabled.
2. Clone this repo

   $ go get github.com/cloudfoundry-incubator/kubo-disaster-recovery-acceptance-tests
   $ cd $GOPATH/src/github.com/cloudfoundry-incubator/kubo-disaster-recovery-acceptance-tests

3. Create an config.json file, for example:

   {
     "run_test_case_deployment": true,
     "run_test_case_etcd_cluster": true,
     "timeout_in_minutes": 5
   }

4. Export CONFIG_PATH to be path to config.json file you just created.
5. Export the following BOSH environment variables
   • BOSH_ENVIRONMENT - URL of BOSH Director which has deployed the CFCR cluster
   • BOSH_CLIENT - BOSH Director username
   • BOSH_CLIENT_SECRET - BOSH Director password
   • BOSH_CA_CERT - BOSH Director's CA cert content
   • BOSH_ALL_PROXY - optional, set the proxy to be used in case the BOSH director is behind a jumpbox
   • BOSH_DEPLOYMENT - name of the CFCR deployment to backup and restore
6. Configure kubectl with a cluster and admin credentials, for example:

   $ kubectl config set-cluster CLUSTER_NAME --server=https://API_IP:8443 --certificate-authority=PATH/TO/CA_CERT --embed-certs=true
   $ kubectl config set-credentials ADMIN_USERNAME --token=ADMIN_PASSWORD
   $ kubectl config set-context NAME --cluster=CLUSTER_NAME --user=ADMIN_USERNAME
   $ kubectl config use-context NAME

7. Export KUBECONFIG to the path to the kubectl config, for example ~/.kube/config.
8. Run acceptance tests

   $ ./scripts/_run_acceptance_tests.sh

Config Variables

• timeout_in_minutes - ginkgo Eventually timeout in minutes, default 5
• run_test_case_<test-case-name> - flag for whether to run a given test case, if omitted defaults to false

Contributing to k-DRATs

k-DRATS runs a collection of test cases against a CFCR cluster.

Test cases should be used for checking that K8s data has been backed up and restored correctly. For example, if two workflows are deployed before bbr director backup, and the workflows are removed after taking the backup. Then after a successful bbr director restore, workflows will be restored back to their original state.

To add extra test cases, create a new test case that implements the TestCase interface.

The methods that need to be implemented are:

• Name() string - should return name of the test case.
• BeforeBackup(Config) - runs before the backup is taken, and should create state in the K8s cluster to be backed up.
• AfterBackup(Config) - runs after the backup is complete but before the restore is started.
• AfterRestore(Config) - runs after the restore is complete, and should assert that the state in the restored K8s cluster matches that created in BeforeBackup(Config).
• Cleanup(Config) - should clean up the state created in the K8s cluster through the test.

Config contains the config for accessing the target K8s. Note: the use of this config is optional, kubectl is already configured to access the target K8s cluster when a test case runs.

Creating a new test case

1. Create a new test case in the testcases package.
2. Add the newly created test case to the list of availableTestCases in acceptance_suite_test.go.