eLife k8s/Flux Production Cluster

EKS cluster name: kubernetes-aws--flux-prod

Use this git repo to control the cluster state (no kubectl or helm cli action needed/wanted).

• Flux will try to apply any yaml file in this repo to the cluster
• HelmController allows use of helm charts
• We currently have three Kustomizations defined: crds, system and deployments (each pointed at the root directory named the same). Only Yaml files found in these folders are loaded, in a dependency order (see "Kustomizations" below)

Cluster infrastructure is defined in builder in the kubernetes-aws section.

Users can configure kubectl for this cluster with:

    aws eks update-kubeconfig --name kubernetes-aws--flux-prod

Dashboards

• Kubernetes Dashboard
• Grafana Dashboards
• Alertmanager

The #cluster-alerts slack channel receives alerts from:

• Alertmanager
• Healthchecks.io (monitors Alertmanager heartbeat)

How to make a change

• Ensure that you have run mise install to get up-to-date dependencies.
• Ensure that you have run make validate and it completed correctly.
• Follow conventional commits for commit messages.
• Add a suffix to the commit message containing the issue e.g. elifesciences/issues#1234
• Look at the GitOps dashboard to see the change being applied.

Adding/Editing Deployments

Kustomizations

Cluster level Kustomizations

• crds: Cluster managed CustomResourceDefinitions.

• system: Cluster services that are not directly serving production usecases. Some infrastructure components needs CRDs to exist before upgrading, so infrastructure kustomization depends on crds kustomization

• deployments: These are the production services. As these all depend on infrastructure to serve traffic correctly, system kustomization is a dependency of this kustomization

• flux tries to apply any .yaml file in the kustomization directories above

• within that root folder, the structure is only used for humans

• namespaces are managed using .yaml files

• flux will always apply the HEAD of master

Each namespace is organised around an application, or an environment for an application, favouring the latter.

Individual Kustomizations

There are a growing number of kustomizations for apps or system that abstract complexity. We can then deploy them with a flux Kustomization object from one of the cluster kustomizations above. These kustomizations are stored in kustomzations/.

Adding Helm Charts

• add a "source" object for the HelmChart (either HelmRepository, GitRepository or Bucket source type)
• add a HelmRelease object, see docs
• Flux can automatically update images in your chart
  • Setup an ImageRepository to query container registry for tags
  • Setup an ImagePolicy to choose what the latest tag is
  • Setup an ImageUpdateAutomation to describe which GitRepository object you want flux to update, and which directory
  • Add a policy marker to tell Flux how to update te yaml files

Debugging

• k8s dash
• Flux logs
• HelmOperator logs
• State of all Canaries

Services available on the Cluster

• traefik (docs)
  • provides SSL termination
  • host entries ending in .elifesciences.org will be added to our zone by ExternalDNS
• cert-manager with letsencrypt (docs/letsencrypt)
  • obtain letsencrypt SSL certs via ingress definitions
• VictoriaMetricsOperator (docs/monitoring-alerting)
• oauth2-proxy (docs/oauth-proxy)
  • limit access to elifesciences github org
• SealedSecrets (docs/sealed-secrets.md)
  • encrypt secrets for safe storage in this repo
• VictoriaLogs
  • Stores logs for services in cluster, is queriable from Grafana as a data source.
• Percona Server for MongoDB operator
  • Used to run a MongoDB cluster, with support for automated backup, reconvery and upgrades.
  • Deployed in "cluster-wide" mode. Each namespace can deploy it's own cluster of pods from the central operator.

Administration

• Upgrading infrastructure services
• Adding a new application team
• Bootstrapping the cluster
• Upgrading EKS/Worker Nodesl
• Debugging/Common Fixes