Skip to content

README.md

Latest commit

 

History

History
212 lines (192 loc) · 20.7 KB

README.md

File metadata and controls

212 lines (192 loc) · 20.7 KB
Raw

EKS-D automated deployment

Below is an example (not OOTB production-grade solution) for an EKS-D automated deployment over zCompute - facilitating cloud integration with dynamic ASG scaling, instance labeling & lifecycle, native load balancing, built-in storage capabilities and optional Kasten K10 as a backup & restore solution.

Known limitations

  • zCompute minimal version is 23.08 running in VSC-mode
  • Upgraded zCompute clouds must have at least one AWS-compatible VolumeType API Alias (io1 / io2 / gp2 / gp3 / sc1 / st1 / standard / sbp1 / sbg1) to be available for provisioning (fresh 23.08 installations have them OOTB)
  • The EKS-D cluster name (set by the environment variable as mentioned below) must be unique for the account
  • The Cluster Autoscaler might also scale-down the control-plane ASG which may affect the ETCD quorom and even brick the cluster, so make sure to use min=max=desired for the masters ASG capacity (default is 1=1=1)

Security concerns

  • The pre-baked EKS-D image modifies the default Ubuntu udev service sandboxing permissions by allowing API calls to be made upon new volume attachment (required for the EBS CSI operation)
  • The deployment will create and use a bastion VM with port 22 (SSH) exposed to the world (and EKS-D nodes with port 22 exposed to the bastion) - you may want to limit this exposure, stop or even terminate the bastion VM post-deployment
  • The deployment will create a public-facing NLB for the control-plane api-server, exposing Kubernetes to the world - you may want to limit this exposure to private networks per the documentation below

zCompute prerequisites

  • Storage:
    • Verify your provisioning-enabled VolumeType aliases - usually this would be gp2 but you can validate it by asking your cloud admin or running the below Symp command via the Zadara toolbox VM using your zCompute account (domain) and credentials:
      volume volume-types list -c name -c alias -c is_provisioning_disabled -c is_default -c state -c health -m grep=Normal
      The EBS CSI will use 'gp2' as the default VolumeType unless specified otherwise via the terraform ebs_csi_volume_type variable in the eksd-terraform project
  • Images:
    • Ubuntu 22.04 image should be imported from the Marketplace to be used for the Bastion VM
    • Zadara's pre-baked EKS-D image should be imported from the Marketplace to be used for the Kubernetes nodes
  • Credentials:
    • Key-pair for the bastion server (either import or create a new one)
    • Key-Pair for the master servers (can be the same)
    • Key-Pair for the worker agents (can be the same)
    • AWS programmatic credentials (access key & secret key) with tenant-admin, AWS MemberFullAccess & IAMFullAccess permissions for the relevant project

All-In-One approach

For a simplified/demo experience, you can use this option to streamline a cluster deployment with a single command - you will get the OOTB default values of a small-sized cluster with a basic CNI (Flannel) and all addons except for Kasten K10. Note this option should not be used for production-grade deployments (for example the default control-plane is not HA), however you may change the default values as mentioned below to use this approach for any cluster configuration.

  • Copy the terraform.tfvars.template file to terraform.tfvars and edit the parameters:
    • api_endpoint - the URL/IP of the zCompute cluster
    • environment - prefix for the various resources to be created (defaults to "k8s")
    • bastion_keyname - the bastion Key-Pair name
    • bastion_keyfile - the bastion Key-Pair private PEM file location
    • bastion_ami - the Ubuntu/CentOS bastion image AMI (AWS ID)
    • bastion_user - depending on the bastion AMI (either ubuntu or centos)
    • eksd_ami - the pre-baked EKS-D image AMI (AWS ID)
    • masters_keyname - the masters Key-Pair name
    • masters_keyfile - the masters Key-Pair private PEM file location
    • workers_keyname - the workers Key-Pair name
    • workers_keyfile - the workers Key-Pair private PEM file location
  • Optional - create a non-default deployment
    • Check the below infra-terraform & eksd-terraform projects for their specific variables and their default values in the respected variables.tf files, or set an environment variable TF_VAR_<variable name>=<value> before your run
    • For example, you can set the ebs_csi_volume_type variable in the eksd-terraform project to something other than 'gp2' per your storage preferences
    • IMPORTANT - for production use-cases, Zadara recommends revising all non-default properties mentioned in the sections below - for example setting 3 master nodes (using masters_count) for control-plane high-availability, setting the external backup properties (like backup_bucket) for control-plane DR capabilities, etc.
  • Run apply-all.sh <access_key> <secret_key> with your access_key & secret_key as the parameters (or set the AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY environment variables before running the script without specifying parameters)
    • The script will take about 10 minutes for a successful minimal deployment of a single master & worker
    • The script can be rerun for re-apply Terraform changes (for example as part of an upgrade procedure)
    • If neccessary, you can destroy all assets and reset everything with the destroy-all.sh script (with the same two credentials parameters/variables)
  • Once completed you will see the kubeconfig content ready for your usage (presented on screen and as a kubeconfig file in the running directory) so you can skip the next two phased approach steps and use it as-is ;-)

Phased approach step 1 - Infrastructure deployment

  • Go to the infra-terraform directory
  • Copy the terraform.auto.tfvars.template file to terraform.auto.tfvars and edit the parameters
    • api_endpoint - the URL/IP of the zCompute cluster
    • cluster_access_key - the tenant admin access key
    • cluster_access_secret_id - the tenant admin secret key
    • bastion_keyname - the Key-Pair for the bastion
    • bastion_ami - the Ubuntu/CentOS bastion image AMI (AWS ID)
    • environment - prefix for the various resources to be created (defaults to "k8s")
  • terraform init - this will initialize Terraform for the environment
  • terraform plan - this will output the changes that Terraform will actually do (resource creation), for example:
    • VPC to hold the entire solution
    • Public subnet for the bastion VM and its corresponding Internet Gateway
    • Private subnet for the Kubernetes nodes VMs
    • Routing tables to accomodate public/private subnets
    • Default Security Group for the VPC as well as RKE2-related one (based on the SG itself)
    • Bastion VM on the public subnet (accessible to the world by default) with access to the private subnet (where the Kubernetes nodes will be located)
    • Network Load Balancer to hold the Kubernetes API Server endpoints - accessible to the world by default, can be hardened for internal-only access if you add the variable expose_k8s_api_publicly = false
    • Elastic IPs for the Bastion as well as the Network Load Balancer
  • terraform apply --auto-approve - this will make the actual changes on the environment
  • Due to current zCompute limitation, you will need to re-apply Terraform again in order to populate some resource tags (resource names, etc.)
  • Terraform will output the relevant information required for the next step - if you lose track of them you can always run terraform output to list them again
  • In the next step you will also be required to provide the NLB's private & public IPs - you can get those from the GUI or by running the get_loadbalancer.sh script as proposed in the terraform output message
  • Note that the subnets' MTU must match the edge network MTU - if there's a mismatch you should adjust both private & public subnets MTUs accordingly via zCompute GUI before continuing

Phased approach step 2 - EKS-D deployment

  • Go to the eksd-terraform directory
  • Copy the terraform.auto.tfvars.template file to terraform.auto.tfvars and edit the parameters
    • Populate the sensitive variables (you may want to pass them at runtime rather than save them)
      • cluster_access_key - admin access key
      • cluster_access_secret_id - admin secret key
    • Paste the previous step's outputs (except the last x_loadbalancer_script entry)
      • api_endpoint - the URL/IP of the zCompute cluster
      • bastion_ip - the bastion VM's public IP
      • environment - Kubernetes cluster name & prefix for various resources (defaults to "k8s")
      • masters_load_balancer_id - the NLB id
      • masters_load_balancer_internal_dns - the NLB internal DNS name
      • private_subnet_id - the private subnet id (in which all of the VMs will be created)
      • public_subnet_id - "the public subnet id (in which the NLB will listen)
      • security_group_id - the security group id to be applied on all VMs
      • vpc_id - the target VPC id
    • Paste the NLB private & public IP addresses (can be fetched by running the get_loadbalancer.sh script)
      • masters_load_balancer_private_ip - the NLB private IP
      • masters_load_balancer_public_ip - the NLB public IP
    • Populate extra mandatory variables
      • eksd_ami_id - the EKS-D AMI (AWS ID)
      • master_key_name - the Key-Pair name for the master VMs (you may reuse the bastion key)
      • worker_key_name - the Key-Pair name for the worker VMs (you may reuse the bastion key)
    • Populate extra optional variables
      • masters_count - the amount of master nodes (minimal is 1, defaulting to 1 but suggested 3 for HA)
      • workers_count - the amount of worder nodes (minimal is 0, defaulting to 1 + 3 more for max ASG size of 4)
      • masters_instance_type - the masters VM size (minimal is z2.large, defaulting to z4.large)
      • masters_instance_type - the workers VM size (minimal is z2.large, defaulting to z8.large)
      • masters_volume_size - the masters disk size (minimal is 30GB, defaulting to 50GB)
      • workers_volume_size - the workers disk size (minimal is 30GB, defaulting to 100GB)
      • cni_provider - choose the CNI from a list of flannel (default), calico or cilium (experimental)
      • ebs_csi_volume_type - the cloud's storage VolumeType (defaulting to gp2)
      • install_ebs_csi - whether to deploy the EBS CSI driver addon (defaulting to true)
      • install_lb_controller - whether to deploy the AWS Load Balancer Controller addon (defaulting to true)
      • install_autoscaler - whether to deploy the Cluster Autoscaler addon (defaulting to true)
      • install_kasten_k10 - whether to deploy the Kasten K10 addon (defaulting to false)
      • backup_access_key_id - external NGOS/S3 user access-key for ETCD backup export
      • backup_secret_access_key - external NGOS/S3 user secret-key for ETCD backup export
      • backup_region - external NGOS/S3 region for ETCD backup export (defaulting to us-east-1)
      • backup_endpoint - external NGOS endpoint for ETCD backup export (not needed for AWS S3)
      • backup_bucket - external NGOS/S3 bucket name for ETCD backup export
      • backup_rotation - maximal number of remote backup files to retain (defaulting to 100)
  • terraform init - this will initialize Terraform for the environment
  • terraform plan - this will output the changes that Terraform will actually do (resource creation), for example:
    • EKS-D master nodes ASG + Launch Configuration
    • Load Balancer target group for the master nodes ASG + publish listener on 6443 for the API server
    • EKS-D worker nodes ASGs + Launch Configuration
    • Tag the existing private & public subnets for Load Balancer controller discovery
  • terraform apply - this will make the actual changes on the environment

Once Terraform is over, you will need to get the kubeconfig file from the first master node - you can use the get_kubeconfig.sh script as mentioned on the terraform output in order to fetch the initial kubeconfig from the first master node (through the bastion) into the project's directory.

Use the kubeconfig to connect to the Kubernetes cluster in the usual way - congratulations on your new cluster :)

OOTB content

Your cluster comes pre-deployed with the below utilities:

  • CCM - using the AWS Cloud Provider, providing you the below abilities:
    • Instances lifecycle updates (Kubernetes will be aware of new/removed Kubernetes nodes)
    • Instances information labeling (Kubernetes will show EC2 Instance information as node labels)
    • LoadBalancer abilities - note this is NLB only, and the AWS Load Balancer Controller addon will override this specification if enabled
      • Add the below annotation for all LoadBalancer specifications:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
      • Add the below annotation for all public-facing NLBs (the default is internal-facing):
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"
  • CNI - either Flannel (default), Calico or Cilium
    • Flannel - basic pod networking abilities, suitable for most use-cases
    • Calico - advanced security (may require further configuration)
    • Cilium - eBPF-based networking with built-in observability (experimental)

Optional addons

As mentioned, your cluster can come pre-deployed with the latest versions (at the time of EKS-D image baking) of the below addons. Alternatively, you may change/delete them via helm after the deployment, or choose to install them by yourself:

  • EBS CSI driver (enabled by default):
    • The ebs-cs StorageClass is pre-configured with the VolumeType and set as the default StorageClass (you may override it with other CSIs)
    • The snapshotting abilities are pre-configured with the ebs-vsc VolumeSnapshotClass (including the Kasten-ready annotation for seamless operability)
    • For self-installation, use the dedicated instructions
  • AWS Load Balancer Controller (enabled by default)
    • For NLB - use the LoadBalancer service per the documentation
      • The latest controller version overrides the built-in LoadBalancer resource, so you just need to add the service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing annotation for internet-facing NLB (as the default is internal)
      • As a known limitation, the controller wouldn't create the relevant security group to the NLB - rather, it will add the relevant rules to the worker node's security group and you can attach this (or another) security group to the NLB via the zCompute GUI, AWS CLI or Symp
    • For ALB - use the Ingress resource per the documentation
      • By default all Ingress resources are internal-facing - if you want your ALB to get a public IP you will have to add the alb.ingress.kubernetes.io/scheme: internet-facing annotation
    • For self-installation, use the dedicated instructions
  • Cluster Autoscaler (enabled by default)
    • The configuration is pre-populated to use the auto-discovery mode based on the pre-populated tags on the worker ASG (k8s.io/cluster-autoscaler/enabled and k8s.io/cluster-autoscaler/<cluster-name>) where cluster-name is the environment variable set on the eksd-terraform project
    • If you opt to use the manual mode - remember to define the specific workers ASG/s name/s and their lower/upper bounds on the autoscalingGroups values
    • For self-installation, use the dedicated instructions
  • Kasten K10 (disabled by default, but internal images are pre-fetched)
    • The deployment assumes the EBS CSI addon is already installed (otherwise k10 will fail to load)
    • The snapshotting ability is enabled OOTB (using the default ebs-vsc VolumeSnapshotClass)
    • The export profile is not set OOTB - you will need to configure it in case you want to export the backups outside of zCompute
    • Keep in mind that k10 is only free up to 5 worker nodes - please consult Kasten's pricing for anything above that
    • For self-installation, use the dedicated instructions

Optional post-deployment DR configuration

Unless configured as part of the eksd-terraform variables, the EKS-D cluster's internal ETCD datastore automated backup procedure (running every 2 hours) will only save the latest backup locally within each master node.

Alternatively, users may configure the Kubernetes secret below in order to dynamically enable/disable backup exports to NGOS/S3 in order to enhance the cluster's DR capabilities in case of a control-plane meltdown:

kubectl create secret generic zadara-backup-export \
    --namespace kube-system \
    --from-literal=backup_access_key_id="<access key>" \
    --from-literal=backup_secret_access_key="<secret key>" \
    --from-literal=backup_region="<bucket region>" \
    --from-literal=backup_endpoint="<NGOS endpoint (not relevant for S3)>" \
    --from-literal=backup_bucket="<bucket name>"

Once set, the periodical ETCD backup procedure within each master node will also export the latest backup into the relevant NGOS/S3 location.

Please note such configuration will override the pre-defined terraform variables-based exteral backup configuration.

Optional BYOI (create your own EKS-D image with Packer)

Only relevant if you wish to bake your own EKS-D image

  • Requires importing the Ubuntu 22.04 image from the Marketplace to be used as the base image for EKS-D
  • Requires a temporary use of the bastion VM (or any other VM on the public subnet) - you will need to use the bastion's private key
  • Requires a local/remote environment with access to the bastion's public IP and AWS access & secret keys to zCompute
  • See the packer project documentation for more details

Optional Zadara CSI usage

Only relevant if you wish to utilize the Zadara CSI and use a dedicated VPSA to persist data from your Kubernetes

  • Requires a dedicated VPSA with one pool and a write-enabled user token (access key) - you will need to provide the key to the Zadara CSI Storage Class configuration
  • Make sure routing is in place and Security Group allows communication between the private subnet and the VPSA
  • Add the Zadara CSI Helm repo:
    helm repo add zadara-csi https://raw.githubusercontent.com/zadarastorage/zadara-csi/release/zadara-csi-helm
  • Deploy the CSI driver chart - see values here and note you may want to disable TLS verification for internal VPSAs, for example:
    helm upgrade --install zadara-csi zadara-csi/zadara-csi --set vpsa.verifyTLS=false
  • Follow the post-deployment Helm notes for the CRDs definitions:
    • VSCStorageClass configuration (see full documentation here)
    • VPSA configuration (there goes the VPSA address & the user's token)
  • Deploy a Storage Class which will point to the VSCStorageClass (you might want to set it as the default storage class for simplicity)
  • Further CSI examples (like how to create a block/filesystem PVC, etc.) can be found here