|
1 | 1 | # Remote Kubernetes (GitOps) |
2 | 2 |
|
3 | | -Use this guide to deploy Metaboost to a remote Kubernetes cluster with Argo CD and a separate GitOps repository. |
| 3 | +Use this guide to deploy Metaboost to a remote Kubernetes cluster with Argo CD and a **separate |
| 4 | +GitOps repository**. |
4 | 5 |
|
5 | | -This repository provides application code and shared base manifests under `infra/k8s/base/`. |
| 6 | +**Working directory:** This runbook assumes you have a local checkout of your GitOps repository that |
| 7 | +contains `apps/`, `argocd/`, `secrets/`, and `scripts/`. Cluster-facing and secret-generation steps |
| 8 | +run from that GitOps repository, not from this Metaboost source repository. |
6 | 9 |
|
7 | | -Recommended model: |
| 10 | +This repository is the source for reusable base manifests under `infra/k8s/base/` and reference |
| 11 | +secret-generator scripts under |
| 12 | +[`infra/k8s/scripts/secret-generators/`](../../../infra/k8s/scripts/secret-generators/INFRA-K8S-SCRIPTS-SECRET-GENERATORS.md). |
8 | 13 |
|
9 | | -- Keep per-environment overlays in your GitOps repository. |
10 | | -- Keep environment-specific env files and manifest patches in your GitOps repository. |
11 | | -- Keep encrypted secrets in your GitOps repository. |
12 | | -- Do not run in-repo k8s env contract/render/drift tooling from this repository. |
13 | | - |
14 | | -## Scope |
| 14 | +## Scope and model |
15 | 15 |
|
16 | 16 | - Metaboost repository: |
17 | | - - Owns app code, Docker build inputs, and shared base manifests. |
18 | | - - Can be referenced from GitOps overlays via remote Kustomize resources. |
| 17 | + - Owns application code, Docker build inputs, and shared Kustomize bases (`infra/k8s/base/*`). |
| 18 | + - Owns reference docs and generator source scripts. |
19 | 19 | - GitOps repository: |
20 | | - - Owns `apps/metaboost-<env>/` overlays (for example **alpha** under `apps/metaboost-alpha/` in your GitOps repo). |
21 | | - - Owns Argo CD `Application` and `AppProject` resources. |
22 | | - - Owns ingress hosts, TLS issuers, ConfigMap/Secret values, and SOPS encrypted secrets. |
| 20 | + - Owns environment overlays (`apps/metaboost-<env>/`). |
| 21 | + - Owns Argo CD `AppProject` and `Application` CRs. |
| 22 | + - Owns ingress hosts, TLS issuer wiring, ConfigMap/Secret values, and encrypted SOPS manifests. |
| 23 | + |
| 24 | +The recommended operating model is: |
| 25 | + |
| 26 | +- Keep all environment-specific values and secrets in your GitOps repository. |
| 27 | +- Pin remote bases by immutable tag/sha. |
| 28 | +- Sync applications in dependency order. |
| 29 | + |
| 30 | +## Defaults |
| 31 | + |
| 32 | +- **Image tags (alpha example):** set app images to `newTag: "X.Y.Z-staging.N"`. |
| 33 | +- **Remote base refs (alpha example):** set Metaboost remote bases to `?ref=X.Y.Z-staging.N`. |
| 34 | +- **Bump together:** every publish bump should update both `?ref=` and `images[].newTag`. |
| 35 | +- **No branch refs:** do not commit moving refs like `?ref=main` or `?ref=develop`. |
| 36 | + |
| 37 | +## Encrypted secrets (GitOps repository) |
| 38 | + |
| 39 | +Workload and registry pull secrets should live **only** in your GitOps repository (commonly under |
| 40 | +`secrets/`). |
| 41 | + |
| 42 | +Required secret names expected by current Metaboost base manifests: |
| 43 | + |
| 44 | +- `metaboost-db-secrets` |
| 45 | +- `metaboost-api-secrets` |
| 46 | +- `metaboost-management-api-secrets` |
| 47 | +- `metaboost-keyvaldb-secrets` |
| 48 | +- `github-registry-secret` |
| 49 | +- optional: `metaboost-mailer-opaque` |
| 50 | + |
| 51 | +Reference generator scripts live in this repository under |
| 52 | +[`infra/k8s/scripts/secret-generators/`](../../../infra/k8s/scripts/secret-generators/INFRA-K8S-SCRIPTS-SECRET-GENERATORS.md). |
| 53 | +Many operators keep synced copies under `metaboost.cc/scripts/secret-generators/` (or equivalent) |
| 54 | +and run them there so outputs land directly in that repo’s `./secrets/` tree. |
| 55 | + |
| 56 | +If your ingress uses cert-manager DNS01 with Cloudflare, use |
| 57 | +[`scripts/infra/sops/create_cloudflare_api_token_secret.sh`](../../../scripts/infra/sops/create_cloudflare_api_token_secret.sh) |
| 58 | +to generate `secrets/cloudflare-api-token-secret.enc.yaml` in your GitOps repository. |
| 59 | + |
| 60 | +## End-to-end command checklist |
| 61 | + |
| 62 | +Set once for your target environment: |
| 63 | + |
| 64 | +```bash |
| 65 | +export GITOPS_REPO_DIR="<absolute-path-to-your-gitops-repository>" |
| 66 | +export KUBE_CONTEXT="<kubectl-context-name>" |
| 67 | +export EXPECTED_SERVER_FRAGMENT="<unique-substring-of-api-server-url>" |
| 68 | +export NAMESPACE="metaboost-alpha" |
| 69 | +export ENV="alpha" |
| 70 | +``` |
| 71 | + |
| 72 | +Prerequisites: `kubectl`, `sops`, and access to the GitOps repo’s `.sops.yaml` keys. |
| 73 | + |
| 74 | +### 1. Hard safety gate (context + API server) |
| 75 | + |
| 76 | +```bash |
| 77 | +current="$(kubectl config current-context)" |
| 78 | +test "${current}" = "${KUBE_CONTEXT}" || { echo "wrong kubectl context"; exit 1; } |
| 79 | +server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')" |
| 80 | +echo "${server}" | grep -q "${EXPECTED_SERVER_FRAGMENT}" || { echo "API server mismatch: ${server}"; exit 1; } |
| 81 | +echo "gate ok: ${server}" |
| 82 | +``` |
| 83 | + |
| 84 | +### 2. Ensure namespace exists |
| 85 | + |
| 86 | +```bash |
| 87 | +cd "${GITOPS_REPO_DIR}" |
| 88 | +kubectl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f - |
| 89 | +``` |
| 90 | + |
| 91 | +### 3. Generate encrypted secrets (GitOps repo) |
| 92 | + |
| 93 | +Use your GitOps repository’s script copies (for example under `./scripts/secret-generators/`): |
| 94 | + |
| 95 | +```bash |
| 96 | +cd "${GITOPS_REPO_DIR}" |
| 97 | +bash ./scripts/secret-generators/create_github_registry_secret.sh "${ENV}" |
| 98 | +bash ./scripts/secret-generators/create_all_secrets_auto_gen.sh "${ENV}" |
| 99 | +bash ./scripts/secret-generators/check_db_secret_contract.sh "${ENV}" |
| 100 | +``` |
23 | 101 |
|
24 | | -## Recommended flow |
| 102 | +If your GitOps repo includes a Metaboost version-contract script (for example |
| 103 | +`scripts/check_metaboost_alpha_version_contract.sh` in `metaboost.cc`), run it before sync. |
25 | 104 |
|
26 | | -1. Update image tags and remote base refs in your GitOps overlays. |
27 | | -2. Edit env values and Kustomize patches directly in your GitOps overlays. |
28 | | -3. Encrypt/update secrets in your GitOps repository (SOPS). Optional helpers live in this monorepo under [`infra/k8s/scripts/secret-generators/`](../../../infra/k8s/scripts/secret-generators/); see [`INFRA-K8S-SCRIPTS-SECRET-GENERATORS.md`](../../../infra/k8s/scripts/secret-generators/INFRA-K8S-SCRIPTS-SECRET-GENERATORS.md). |
29 | | -4. Run `kubectl kustomize` for each overlay in your GitOps repository. |
30 | | -5. Commit and push GitOps changes. |
31 | | -6. Sync Argo CD applications in dependency order. |
| 105 | +### 4. Server dry-run secret apply, then apply for real |
32 | 106 |
|
33 | | -## Environment values and manifests |
| 107 | +```bash |
| 108 | +cd "${GITOPS_REPO_DIR}" |
| 109 | +test -f .sops.yaml || { echo "need .sops.yaml at gitops root"; exit 1; } |
34 | 110 |
|
35 | | -Maintain these directly in your GitOps repository: |
| 111 | +for f in ./secrets/metaboost-"${ENV}"-*.enc.yaml; do |
| 112 | + test -e "${f}" || continue |
| 113 | + sops -d "${f}" | kubectl apply --dry-run=server -f - |
| 114 | +done |
36 | 115 |
|
37 | | -- ConfigMap env fragments and `configMapGenerator` input files. |
38 | | -- Secret manifests (encrypted) and any secret projection patches. |
39 | | -- Deployment/Service/Ingress port patches. |
40 | | -- Hostname/CORS/cookie/public URL values. |
| 116 | +if test -f ./secrets/github-registry-secret.enc.yaml; then |
| 117 | + sops -d ./secrets/github-registry-secret.enc.yaml | kubectl apply --dry-run=server -f - |
| 118 | +fi |
41 | 119 |
|
42 | | -If your ingress uses cert-manager with Cloudflare DNS01, generate an encrypted SOPS manifest for the |
43 | | -token using the reference script |
44 | | -[`scripts/infra/sops/create_cloudflare_api_token_secret.sh`](../../../scripts/infra/sops/create_cloudflare_api_token_secret.sh). |
45 | | -Copy the helper into your GitOps checkout next to `.sops.yaml` before running; default output is |
46 | | -`secrets/cloudflare-api-token-secret.enc.yaml`. Use a Cloudflare API token with `Zone - DNS - Edit` |
47 | | -and `Zone - Zone - Read`, scoped only to required zones, and ensure cert-manager reads Secret |
48 | | -`cloudflare-api-token-secret` from namespace `cert-manager` with key `api-token`. |
| 120 | +for f in ./secrets/metaboost-"${ENV}"-*.enc.yaml; do |
| 121 | + test -e "${f}" || continue |
| 122 | + sops -d "${f}" | kubectl apply -f - |
| 123 | +done |
| 124 | + |
| 125 | +if test -f ./secrets/github-registry-secret.enc.yaml; then |
| 126 | + sops -d ./secrets/github-registry-secret.enc.yaml | kubectl apply -f - |
| 127 | +fi |
| 128 | +``` |
| 129 | + |
| 130 | +### 5. Update GitOps overlay env and patches |
| 131 | + |
| 132 | +Maintain these in your GitOps repository: |
| 133 | + |
| 134 | +- `apps/metaboost-<env>/<component>/source/*.env` |
| 135 | +- deployment/service/ingress patches |
| 136 | +- hostname/CORS/cookie/public URL values |
| 137 | +- TLS issuer annotations and cert-manager integration details |
| 138 | + |
| 139 | +### 6. Local kustomize compile for every overlay |
| 140 | + |
| 141 | +```bash |
| 142 | +cd "${GITOPS_REPO_DIR}" |
| 143 | +for c in common db keyvaldb ops api management-api web management-web; do |
| 144 | + kubectl kustomize "apps/metaboost-${ENV}/${c}" --load-restrictor LoadRestrictionsNone >/dev/null |
| 145 | + echo "ok apps/metaboost-${ENV}/${c}" |
| 146 | +done |
| 147 | +``` |
| 148 | + |
| 149 | +### 7. Apply Argo `AppProject` and `Application` manifests |
| 150 | + |
| 151 | +```bash |
| 152 | +cd "${GITOPS_REPO_DIR}" |
| 153 | +kubectl apply --dry-run=server -f argocd/apps/project-metaboost.yaml |
| 154 | +kubectl apply --dry-run=server -f "argocd/metaboost-${ENV}/" |
| 155 | +kubectl apply -f argocd/apps/project-metaboost.yaml |
| 156 | +kubectl apply -f "argocd/metaboost-${ENV}/" |
| 157 | +``` |
| 158 | + |
| 159 | +### 8. Sync applications in dependency order and verify |
| 160 | + |
| 161 | +Recommended sync order: |
| 162 | + |
| 163 | +1. `metaboost-<env>-common` |
| 164 | +2. `metaboost-<env>-db` |
| 165 | +3. `metaboost-<env>-keyvaldb` |
| 166 | +4. `metaboost-<env>-ops` |
| 167 | +5. `metaboost-<env>-api` |
| 168 | +6. `metaboost-<env>-management-api` |
| 169 | +7. `metaboost-<env>-web` |
| 170 | +8. `metaboost-<env>-management-web` |
| 171 | + |
| 172 | +Then verify: |
| 173 | + |
| 174 | +```bash |
| 175 | +kubectl -n "${NAMESPACE}" get pods |
| 176 | +kubectl -n "${NAMESPACE}" get svc,ingress |
| 177 | +kubectl -n argocd get applications |
| 178 | +``` |
| 179 | + |
| 180 | +## GitOps overlay contract |
| 181 | + |
| 182 | +- Every deployed overlay uses immutable refs for remote Metaboost bases. |
| 183 | +- Every Metaboost workload image tag in overlays matches the pinned release ref. |
| 184 | +- ConfigMap env fragments stay in overlay `source/*.env` files; secrets stay in encrypted manifests. |
| 185 | +- Keep app-specific hostnames and URL/cookie/CORS values in GitOps overlay files, not in this repo. |
| 186 | + |
| 187 | +## Argo CD source contract |
| 188 | + |
| 189 | +- `AppProject` must allow both your GitOps repo and `https://github.com/podverse/metaboost.git` in |
| 190 | + `sourceRepos` so remote Kustomize URLs resolve. |
| 191 | +- `Application.spec.source.path` should reference overlay paths under |
| 192 | + `apps/metaboost-<env>/<component>`. |
| 193 | +- Use one tracked branch in GitOps (commonly `main`) and separate environment paths |
| 194 | + (`metaboost-alpha`, `metaboost-beta`, `metaboost-prod`) instead of branch-per-env. |
49 | 195 |
|
50 | 196 | ## Verification checklist |
51 | 197 |
|
52 | | -- Pods are healthy in target namespace. |
53 | | -- Services and ingress are present and routed correctly. |
54 | | -- Public API/web endpoints respond as expected. |
55 | | -- Argo CD shows synced and healthy applications. |
| 198 | +- Argo applications are **Synced** and **Healthy**. |
| 199 | +- Namespace pods are running and stable (no restart loops). |
| 200 | +- DB and Valkey are reachable by API and management-api workloads. |
| 201 | +- Public web/API and management hostnames route correctly via ingress. |
| 202 | +- API health checks return expected status at your deployed paths. |
56 | 203 |
|
57 | | -## Related |
| 204 | +## Related docs |
58 | 205 |
|
59 | 206 | - [ARGOCD-GITOPS-METABOOST.md](ARGOCD-GITOPS-METABOOST.md) |
60 | 207 | - [GITOPS-CUTOVER-STAGING-CHECKLIST.md](GITOPS-CUTOVER-STAGING-CHECKLIST.md) |
61 | 208 | - [K8S-ENV-RENDER.md](K8S-ENV-RENDER.md) |
62 | 209 | - [infra/k8s/INFRA-K8S.md](../../../infra/k8s/INFRA-K8S.md) |
| 210 | + |
| 211 | +Environment-specific alpha details (example GitOps layout, hostnames, and scripts) are documented in |
| 212 | +`metaboost.cc/docs/k8s/metaboost-alpha/README.md`. |
| 213 | + |
| 214 | +## Documentation guardrails (must pass) |
| 215 | + |
| 216 | +When updating this runbook: |
| 217 | + |
| 218 | +- Keep the command checklist executable from a GitOps repository root. |
| 219 | +- Keep required secret names aligned with current `infra/k8s/base/*` manifests. |
| 220 | +- Keep the sync order aligned with actual component dependencies. |
| 221 | +- Keep links synchronized with: |
| 222 | + - `docs/development/k8s/ARGOCD-GITOPS-METABOOST.md` |
| 223 | + - `docs/development/k8s/GITOPS-CUTOVER-STAGING-CHECKLIST.md` |
| 224 | + - environment-specific GitOps docs (for example `metaboost.cc/docs/k8s/metaboost-alpha/README.md`) |
0 commit comments