|
| 1 | +== Secrets Management in Validated Patterns |
| 2 | + |
| 3 | +[#esoVault] |
| 4 | +== ESO and Vault |
| 5 | + |
| 6 | +IMPORTANT: The Validated Patterns team settled on using **references to secrets** |
| 7 | + |
| 8 | +We currently implement GitOps + Secrets via link:https://external-secrets.io/v0.7.0/[External Secrets Operator] and link:https://www.vaultproject.io/[HashiCorp Vault] |
| 9 | +The main reasons for this choice at the time were the following: |
| 10 | +* Sealed Secret approach carries additional risks |
| 11 | +* ESO allows us a certain level of independence from the Secrets Management System |
| 12 | +* HashiCorp Vault is currently the most popular Secret Management Systems |
| 13 | +* Advised by our security team |
| 14 | +* CyberArk leaders (masters) cannot be hosted on OpenShift |
| 15 | +* Secret injection is incompatible with Pods owned by Operators |
| 16 | + |
| 17 | + |
| 18 | +[#approaches] |
| 19 | +== Approaches |
| 20 | + |
| 21 | +* There are two fundamental approaches to manage secret material within a GitOps context |
| 22 | +* **Encrypted Secrets** stored inside Git repositories |
| 23 | +* **References to Secrets** stored inside Git repositories and the actual secret is stored somewhere else |
| 24 | +
|
| 25 | +[#encryptedSecrets] |
| 26 | +=== Encrypted Secrets |
| 27 | + |
| 28 | +* Secrets are stored in an encrypted form inside the Git repository |
| 29 | +* Automation and helper tools do the decryption to create Kubernetes secrets from them |
| 30 | +* A number of projects exist that implement this approach: |
| 31 | +** link:https://github.com/bitnami-labs/sealed-secrets[Sealed Secrets] |
| 32 | +** link:https://github.com/mozilla/sops[Mozilla’s Secret OPerationS (SOPS)] |
| 33 | +** Other smaller ones (link:https://github.com/kapicorp/tesoro[Tesoro], link:https://github.com/Soluto/kamus[Kamus]) |
| 34 | +
|
| 35 | +These projects provide an easy way to encrypt and decrypt the secrets, and make sure that only authorized users can access them. |
| 36 | +The encryption method and key management should be considered carefully as they are important to ensure the security of the secrets. |
| 37 | + |
| 38 | +IMPORTANT: You should evaluate and choose the right project that fits your specific use case and requirements. |
| 39 | + |
| 40 | +[#secretReferences] |
| 41 | +=== References to Secrets |
| 42 | + |
| 43 | +This approach requires two main parts: |
| 44 | +* A Secret Management System (link:https://www.vaultproject.io/[HashiCorp Vault], link:https://www.conjur.org/[Conjur], link:https://lyft.github.io/confidant/[Confidant]) |
| 45 | +* A controller that retrieves the secret from the Secret Management System and translates it to a Kubernetes Secret or injects it into the Pod directly (link:https://external-secrets.io/v0.7.0/[External Secrets Operator], link:https://github.com/kubernetes-sigs/secrets-store-csi-driver[Kubernetes Secrets Store CSI Driver], link:https://developer.hashicorp.com/vault/docs/platform/k8s/injector[Vault Agent Injector]) |
| 46 | +* Secrets are uploaded/created directly into the Secret Management System |
| 47 | +* References to secrets are stored in Git |
| 48 | +* A controller reads the references to secrets and translates them into Kubernetes secrets |
| 49 | + |
| 50 | +IMPORTANT: HashiCorp Vault runs only on the HUB |
| 51 | + |
| 52 | +[#vault] |
| 53 | +* The Vault has multiple kubernetes backends configured by the unsealVault Cron Job (one for each cluster) |
| 54 | +* There is one ESO instance for each cluster (ESO is currently unsupported) |
| 55 | +* Each ESO instance logs into the HashiCorp Vault using a specific login path that was configured with the credentials of that instance |
| 56 | +* Vault will authenticate the requests by talking to the API endpoints of the clusters where the login request originated from |
| 57 | + |
| 58 | +[#eso] |
| 59 | + |
| 60 | +* At a high-level the External Secrets Operator reads secrets from the Vault and converts them into Kubernetes Secrets |
| 61 | +* To populate the Vault with our secret material we use `make load-secrets` |
| 62 | +* Normally a user would create ~/values-secret-<patternname>.yaml following the contents of values-secret.yaml.template in the pattern’s git repository |
| 63 | +* Running `./pattern.sh make load-secrets` will populate the Vault with our secrets |
| 64 | +
|
| 65 | +NOTE: Provisioning the vault can be complex, so it is taken care of by the framework |
| 66 | + |
| 67 | +* Vault unsealing and configuration happens in the **imperative** namespace with a CronJob |
| 68 | +
|
| 69 | +NOTE: The Unseal vault cron job is idempotent and runs every five minutes on the hub only |
| 70 | + |
| 71 | +image::unseal-vault-cronjob.png[] |
| 72 | + |
| 73 | +* The **vaultkeys** secret containing the keys to unseal the vault and the vault root token that can be used to login to the vault’s UI is created |
| 74 | +
|
| 75 | +IMPORTANT: It is **strongly** recommended that the **vaultkeys** secret in the **imperative** namespace is exported to a safe space and removed. It contains the keys needed to unseal the vault whenever it gets restarted |
| 76 | + |
| 77 | +[#vaultconcepts] |
| 78 | +=== Vault Concepts |
| 79 | + |
| 80 | +* Secrets are stored in **Vault** in specific **paths** |
| 81 | +* **secret/global/config-demo** is the **path** |
| 82 | +* Each **path** can have multiple attributes |
| 83 | +* **secret** is one attribute at this path |
| 84 | +* In this case secret is autogenerated by default inside the vault directly |
| 85 | +
|
| 86 | +[.INFORMATION] |
| 87 | +==== |
| 88 | +By default there are three paths that can be used |
| 89 | + |
| 90 | +* **secret/global/*** accessible by the hub and all managed clusters |
| 91 | +* **secret/hub/*** accessible only by the hub cluster |
| 92 | +* **secret/<fqdn of managed cluster>/*** accessible only by the fqdn-defined cluster |
| 93 | +==== |
| 94 | +
|
| 95 | +TIP: You can think of a path as being a single dictionary with multiple keys. An external secret looks for a path and then has a templating system to extract keys and put them in kubernetes secrets. |
| 96 | + |
| 97 | +[source,yaml] |
| 98 | +---- |
| 99 | +secrets: |
| 100 | + - name: config-demo |
| 101 | + vaultPrefixes: |
| 102 | + - global |
| 103 | + fields: |
| 104 | + - name: secret |
| 105 | + onMissingValue: generate |
| 106 | + vaultPolicy: validatedPatternDefaultPolicy |
| 107 | +---- |
| 108 | + |
| 109 | +From the vault interface perspective: |
| 110 | + |
| 111 | +image::global-configdemo-vault.png[] |
0 commit comments