Setup: GitOps Promoter demo on GCP

This guide walks through first-time provisioning and configuration. If something fails, see DEBUGGING.md.

The repository bootstraps a demo environment for GitOps Promoter on Google Cloud. It uses:

GKE for Kubernetes
OpenTofu (tofu) for cluster/network provisioning (same *.tf as Terraform; HashiCorp Terraform 1.6+ also works if you substitute terraform for tofu)
Argo CD with an App-of-Apps pattern
Single-source Argo CD Applications only
Umbrella Helm charts stored in this repository
Your DNS provider for public DNS
cert-manager + ingress-nginx for HTTPS
Sealed Secrets for secret delivery

What this repository bootstraps

The initial bootstrap commit is focused on cluster foundations:

Argo CD (embedded Dex + GitHub connector in Helm values; OAuth credentials via Sealed Secret)
cert-manager
ingress-nginx
Sealed Secrets
GitOps Promoter
base namespaces and RBAC
an ACME ClusterIssuer

Later commits can add:

charts/gitops-promoter/templates/github-app-credentials.sealed.yaml (GitHub App PEM) and edits to promoter-config-github/scm-provider.yaml / git-repository.yaml (§12); for the parallel GitLab guestbook project see docs/gitlab-demo-setup.md
GitOps Promoter ScmProvider, GitRepository, and PromotionStrategy resources (already in repo; need real IDs and secret)
sealed repository-write, Git webhook HMAC, and Dex OAuth credentials under charts/argocd/templates/*.sealed.yaml (same Application/argocd umbrella chart; §8 / §9), plus env/dev, env/e2e, env/prd and env/dev-next, env/e2e-next, env/prd-next so the hydrator and GitOps Promoter can run
monitoring and dashboards

Repository layout

Paths you will touch when cloning, forking, or extending this demo:

apps/: bootstrap Application/root-app only; child apps are Helm templates in charts/apps/ (single repoURL in charts/apps/values.yaml — keep apps/root-app.yaml spec.source.repoURL the same)
demo-apps/guestbook/: minimal in-tree Helm chart (Deployment + Service, gcr.io/google-samples/gb-frontend:v5). Guestbook **Application**s use source hydration: Argo reads demo-apps/guestbook on HEAD, writes rendered YAML under hydrated/guestbook-{dev,e2e,prd} on each env’s env/<env>-next branch, and syncs from env/<env> after GitOps Promoter merges env/<env>-next → env/<env>. Per-env replica counts use demo-apps/guestbook/env/{dev,e2e,prd}/values.yaml
charts/: umbrella charts and Helm values (each chart may include Chart.lock from helm dependency build). Argo CD–related SealedSecret manifests live in charts/argocd/templates/ (Dex OAuth, Git webhook HMAC, hydrator repository-write); GitOps Promoter’s GitHub App PEM is charts/gitops-promoter/templates/github-app-credentials.sealed.yaml, applied with the gitops-promoter Application. charts/monitoring/ wraps kube-prometheus-stack (Prometheus Operator, Prometheus, Alertmanager, Grafana) into namespace monitoring (Application/monitoring, sync wave 1).
manifests/: raw Kubernetes manifests applied by Argo CD (top-level YAML is synced by demo-config into gitops-promoter, excluding demo-churn/**). Application/demo-churn syncs manifests/demo-churn/ (recursive) into argocd: github/ CronJob bumps demoChurn.lastBumped on the GitHub bootstrap repo (GitHub App, same hydrator write Secret); gitlab/ CronJob bumps the same field on the GitLab guestbook repo (project access token).
promoter-config-github/, promoter-config-gitlab/: GitOps Promoter CRs applied with Application/promoter-config, which uses multiple sources to sync both directories into gitops-promoter. Each tree has its own ArgoCDCommitStatus and TimedCommitStatus, scoped with demos.gitops-promoter.dev/instance: github|gitlab so controllers do not mix the two guestbook fleets (Argo CD commit status controller, timed, gating).
infra/gcp/terraform/: OpenTofu/Terraform HCL for GCP networking and GKE
infra/gcp/check-prereqs.sh: local environment check script
infra/gcp/get-ingress-lb-ip.sh: print ingress-nginx load balancer IP for DNS A records
docs/: architecture notes

Conventions

Argo CD applications use single-source spec.source, not multi-source apps, except the guestbook env **Application**s, which use spec.sourceHydrator so GitOps Promoter can gate on hydrated branches (source hydrator).
Application/root-app and every child app rendered from charts/apps/templates/ use automated sync with prune: true and selfHeal: true, and metadata.finalizers including resources-finalizer.argocd.argoproj.io, so deleting an Application cascades removal of its managed resources (Argo CD deletion finalizer).
Helm deployments come from in-repo umbrella charts.
OpenTofu inputs for GCP live in infra/gcp/terraform/terraform.tfvars (committed in this demo repo; forks should replace project_id, project_name, and billing_account with their own values).
GitOps first: change the cluster by committing to this repository and letting Argo CD sync. Avoid kubectl apply, kubectl patch, or ad-hoc edits to workloads except in a real break-glass situation (for example, Argo CD cannot reconcile and you need a one-time repair).
After any break-glass change: update the matching manifests or Helm values here and push before you consider the incident closed, so the next sync does not fight the cluster or reintroduce the failure.
Bootstrap exception: the very first Argo CD install still uses a one-time helm template | kubectl apply from charts/argocd (see §7); everything after that should flow from Git. One-off cluster repairs are covered in DEBUGGING.md.

Prerequisites

Install these tools before starting:

git
gcloud
kubectl
tofu (OpenTofu CLI), or terraform 1.6+ if you prefer HashiCorp’s binary
helm
kubeseal (same major line as the in-cluster Sealed Secrets controller) when you create sealed manifests locally

You also need:

a GCP account with billing enabled
a GCP project for the demo
a domain you can manage in your DNS provider
a GitHub repository to host this repo

1. Clone the repository

git clone https://github.com/<your-github-owner>/gitops-promoter-demo.git
cd gitops-promoter-demo

2. Customize repository and domain references

Before provisioning, replace the placeholders and personal org references in the repo.

At minimum, update:

apps/root-app.yaml and charts/apps/values.yaml — set the same Git repoURL in both (root-app is where Argo CD clones from; the chart value drives every child Application)
charts/argocd/values.yaml
charts/gitops-promoter/values.yaml
charts/monitoring/values.yaml
promoter-config-github/git-repository.yaml
promoter-config-github/scm-provider.yaml
(optional) promoter-config-gitlab/* and charts/apps/values.yaml guestbookGitLabRepoURL — see docs/gitlab-demo-setup.md

Things you will almost certainly change:

GitHub owner/repository URLs
Argo CD RBAC (policy.csv) and Dex GitHub connector (orgs/teams limit GitHub login to argoproj-labs / gitops-promoter-approvers; users.anonymous.enabled allows readonly without login; policy.csv maps that team → admin)
public hostnames
GitOps Promoter GitHub owner/repo references
secret names once you introduce real credentials

If you are using your own domain instead of gitops-promoter.dev, update:

demo.<your-domain>
promoter-webhook.<your-domain>
grafana.<your-domain>
charts/gitops-promoter/values.yaml — in controllerConfiguration.pullRequest.template.description, the urlParse host must match your public Argo CD URL (same origin as demo.<your-domain>) so promotion PRs link back to the GitOps Promoter extension (upstream example)

3. Authenticate to Google Cloud

Log into GCP with the same Google identity that will own this demo cluster, and set up Application Default Credentials (ADC) for OpenTofu/Terraform. If you use several accounts on one laptop (for example a personal account and a dedicated demo Gmail), be explicit—otherwise gcloud and ADC can keep using an old account and you will provision or query the wrong project.

3.1 Dedicated Google account (clean slate, recommended)

Use this flow when the demo lives under a new Gmail (or any identity) that should not touch your existing GCP projects.

gcloud auth login <your-demo-google-account>
gcloud config set account <your-demo-google-account>

Detach the CLI default project if it still points at something else (optional but avoids confusion):
```
gcloud config unset project
```
Point ADC at the same user (OpenTofu uses ADC, not the gcloud account alone):
```
gcloud auth application-default login
```
Confirm billing is visible to this account (you need an OPEN billing account ID for terraform.tfvars):
```
gcloud auth list
gcloud billing accounts list
```
You do not need an existing GCP project before the first tofu apply—this repository’s stack always creates the project (see project_id in terraform.tfvars). After apply succeeds, point gcloud and ADC at that project for follow-on commands:
```
gcloud config set project <project_id from terraform.tfvars>
gcloud auth application-default set-quota-project <project_id from terraform.tfvars>
```

4. Prepare variables (`terraform.tfvars`)

Edit infra/gcp/terraform/terraform.tfvars in the repository (or copy from terraform.tfvars.example if you are starting a new fork without the checked-in file).

Pick a globally unique project_id (6–30 characters; lowercase letters, digits, hyphens). It must not reuse an old demo project id from another account.

Set only:

project_id — id for the new project OpenTofu creates
project_name — display name in the Cloud Console
billing_account — OPEN billing account id from gcloud billing accounts list

Region, cluster name, VPC CIDRs, and node pool sizing are locals in main.tf; change them there if you need a different region, larger machines, or higher quotas.

5. Provision the GKE cluster

From the repository root (examples use tofu; run the same commands with terraform if you use HashiCorp Terraform):

cd infra/gcp/terraform
tofu init
tofu plan -out tfplan
tofu apply tfplan
cd ../../..

This stack creates:

a new GCP project (linked to billing_account)
required GCP APIs
a custom VPC
a subnet with pod/service secondary ranges
a regional GKE cluster
a managed node pool with autoscaling

6. Fetch kubeconfig and verify `kubectl`

Use the same Google identity that has access to the GKE cluster (the account you use in the Cloud Console).

6.1 Sign in with the Google Cloud CLI

If you are not already logged in, or your tokens expired:

gcloud auth login

If you use several Google accounts on one machine, sign in to the one that should touch this cluster (replace the address with yours):

gcloud auth login <your-google-account>

Confirm the account you intend to use:

gcloud auth list
gcloud config set account <your-google-account>

6.2 Point `gcloud` at the right project

gcloud config set project <project-id>

To discover cluster name and location (regional clusters use --region; zonal clusters use --zone):

gcloud container clusters list --project <project-id>

6.3 Merge cluster credentials into kubeconfig

For a regional cluster:

gcloud container clusters get-credentials <cluster-name> \
  --region <region> \
  --project <project-id>

For a zonal cluster, use --zone <zone> instead of --region.

This updates your kubeconfig and sets the current context to that cluster.

6.4 Application Default Credentials (optional but recommended)

If kubectl or other tools warn that the quota project on Application Default Credentials does not match your GCP project, align it:

gcloud auth application-default set-quota-project <project-id>

(You may have already run gcloud auth application-default login in section 3 for OpenTofu/Terraform; the quota project can still be set separately.)

6.5 Verify access

kubectl config current-context
kubectl get nodes

On first use, GKE may install a matching kubectl client version automatically. If authentication fails, install the plugin and retry:

gcloud components install gke-gcloud-auth-plugin

7. Bootstrap Argo CD once

Install Argo CD once from this repository’s Helm chart (same version and values as the argocd Application), then hand off to GitOps. Server-side apply avoids CRD last-applied-configuration size limits and matches how large manifests are applied safely.

Sealed Secrets CRD first. The charts/argocd umbrella ships SealedSecret resources under charts/argocd/templates/. The API server rejects them until the Sealed Secrets CRD exists. Install the controller once with the same chart and release name Argo CD will manage later (Application/sealed-secrets → sealed-secrets in kube-system):

cd charts/sealed-secrets
helm dependency build
helm upgrade --install sealed-secrets . -n kube-system --create-namespace
kubectl rollout status deployment/sealed-secrets -n kube-system --timeout=120s
cd ../..

From the repository root, with helm and kubectl configured for the cluster:

kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -
cd charts/argocd
helm dependency build
helm template argocd . -f values.yaml --namespace argocd \
  | kubectl apply --server-side --force-conflicts --field-manager=argocd-bootstrap -f -
cd ../..

The release name argocd must match the Argo CD Helm release the argocd Application expects so labels and selectors stay consistent.

8. Hand off to GitOps

Apply the App-of-Apps root application:

kubectl apply -f apps/root-app.yaml

That causes Argo CD to sync the charts/apps Helm chart and reconcile all child Application objects from charts/apps/templates/.

Git webhooks (sync soon after you push)

By default Argo CD polls Git about every three minutes. To refresh as soon as GitHub (or another provider) receives your push, expose the Argo CD API over HTTPS and register a repository webhook that calls Argo CD’s /api/webhook endpoint. See the upstream Git webhook configuration guide for full detail and other SCMs (GitLab, Bitbucket, Azure DevOps, Gogs).

Prerequisites

DNS and TLS for your Argo CD hostname work (this demo: https://demo.<your-domain>).
GitHub (or your host) can reach that URL from the public internet.

1. Choose a shared secret

Generate a random string (example):

openssl rand -base64 32

You will use the same value in GitHub and in the cluster.

2. Add the webhook in GitHub

In the GitHub repository that Argo CD reads (your fork of this demo repo, or any repo backing an Application):

Settings → Webhooks → Add webhook
Payload URL: https://demo.<your-domain>/api/webhook (use your real Argo CD host; path must be /api/webhook)
Content type: application/json (required; the default form encoding is not supported)
Secret: paste the value from step 1
Events: enable Just the push event (or restrict to pushes only)

Save the webhook. GitHub may show a failed delivery until the sealed manifest in the next section is synced.

Alternative: GitHub App webhook (no per-repo webhook). App webhooks are not the same as Repository → Webhooks; you configure them on the GitHub App itself (e.g. GitHub → Settings → Developer settings → GitHub Apps → your app, or the org’s Settings → GitHub Apps → your app → Edit). There, set Webhook URL to https://demo.<your-domain>/api/webhook, set Webhook secret to the same random string you will seal into argocd-github-webhook, and under Subscribe to events enable Push only for Argo CD’s purposes. Ping deliveries during setup are fine; do not add Pull request, Issues, Workflow runs, etc. for this URL—Argo CD’s /api/webhook handler only parses a small set of events (GitHub push / ping). Extra event types produce 400 with body Webhook processing failed: event not defined to be parsed. Argo CD verifies X-Hub-Signature-256 for App deliveries the same way as for classic repository webhooks. You still seal that shared secret in the next step.

3. Seal the shared secret and commit it (Sealed Secrets only)

Helm in this repo sets webhook.github.secret on argocd-secret to the indirection string $argocd-github-webhook:githubWebhookSecret. Argo CD resolves that at runtime from a normal Secret named argocd-github-webhook in argocd, as in the upstream webhook “Alternative” docs. That Secret must have label app.kubernetes.io/part-of: argocd and data key githubWebhookSecret holding the same value you configured in GitHub.

Where to put the sealed manifest (this repo: charts/argocd/templates/)

From the repository root, run kubeseal (replace the placeholder secret). The Bitnami chart in charts/sealed-secrets exposes the controller as Service sealed-secrets in kube-system, not the kubeseal default sealed-secrets-controller, so the controller flags are required.

kubectl create secret generic argocd-github-webhook -n argocd \
  --from-literal=githubWebhookSecret='PASTE_THE_SAME_SECRET_AS_GITHUB' \
  --dry-run=client -o yaml \
  | kubectl label --local --dry-run=client -f - app.kubernetes.io/part-of=argocd -o yaml \
  | kubeseal \
      --controller-name sealed-secrets \
      --controller-namespace kube-system \
      -o yaml -n argocd \
      -w charts/argocd/templates/argocd-github-webhook.sealed.yaml

Commit charts/argocd/templates/argocd-github-webhook.sealed.yaml and push to the branch root-app tracks.

The Application/argocd umbrella chart includes that template (sync wave -5 on the SealedSecret so it applies before the rest of the release). After sync, the decoded Secret appears without kubectl apply and GitHub deliveries can verify.

If you omit the webhook secret entirely, hooks can still trigger a refresh, but Argo CD cannot verify the sender; for a public URL, configuring the secret is strongly recommended (see the upstream docs).

4. Verify

Push a trivial commit to the tracked branch. In the Argo CD UI, the Application should move to Refreshing quickly instead of waiting for the poll interval. In GitHub, open the webhook’s Recent Deliveries and confirm 200 responses.

Source hydrator credentials (push for in-tree guestbook)

The guestbook **Application**s use Argo CD’s source hydrator: Argo renders the in-tree chart under demo-apps/guestbook, pushes rendered manifests to env/<env>-next branches in this repository, and syncs the cluster from env/<env>. That matches GitOps Promoter’s hard-coded *-next convention (getting started).

Keeping the chart in-tree avoids a second GitHub repo: the hydrator’s dry source and hydrated output both live in gitops-promoter-demo (Argo does not support hydrating into a different repo yet — issue #22719).

This demo enables the hydrator in charts/argocd/values.yaml (commitServer.enabled: true and configs.params.hydrator.enabled: "true"). After the next Helm sync of Argo CD, the application controller and API server pick up argocd-cmd-params-cm.

If this repository is public, Argo CD can clone the dry chart without a repository credential; you still need a repository-write Secret so the hydrator can push. Use a GitHub App with Contents read/write—typically the same installation as GitOps Promoter’s ScmProvider (§12). The data keys are the usual Argo CD repository fields (type, url, githubAppID, githubAppPrivateKey, optional githubAppInstallationID).

If the repository is private, add a second Secret with argocd.argoproj.io/secret-type: repository for the same URL (see the upstream hydrator doc’s pull/push pair).

From the repository root, seal the write secret into charts/argocd/templates/ (same kubeseal controller flags as §8 Git webhook / §9 Dex). Replace the App ID, optional installation ID, PEM path, and repo URL.

kubectl create secret generic argocd-repo-gitops-promoter-write -n argocd \
  --from-literal=type=git \
  --from-literal=url='https://github.com/<your-github-owner>/gitops-promoter-demo' \
  --from-literal=githubAppID='YOUR_GITHUB_APP_ID' \
  --from-file=githubAppPrivateKey=/path/to/your-github-app.private-key.pem \
  --dry-run=client -o yaml \
  | kubectl label --local --dry-run=client -f - argocd.argoproj.io/secret-type=repository-write -o yaml \
  | kubectl label --local --dry-run=client -f - app.kubernetes.io/part-of=argocd -o yaml \
  | kubeseal \
      --controller-name sealed-secrets \
      --controller-namespace kube-system \
      -o yaml -n argocd \
      -w charts/argocd/templates/argocd-repo-gitops-promoter-write.sealed.yaml

Commit that sealed file. It ships with the Application/argocd umbrella (same sync ordering as the other SealedSecret templates). Until it exists, hydration cannot push and the guestbook applications will not reach a healthy sync.

Promoter demo churn (CronJob)

Application/demo-churn (from charts/apps, wave 6) syncs manifests/demo-churn/ recursively into argocd. The GitHub CronJob under github/ runs every 15 minutes, installs PyJWT + requests, and uses Secret/argocd-repo-gitops-promoter-write to call the GitHub Contents API, patching demoChurn.lastBumped in demo-apps/guestbook/values.yaml on GitHub (pod annotations for guestbook-github-* after hydration). The GitLab CronJob under gitlab/ uses Secret/gitlab-demo-churn-credentials (a SealedSecret under charts/argocd/templates/, applied by Application/argocd) and the GitLab Repository Files API; see docs/gitlab-demo-setup.md.

If githubAppInstallationID is omitted from the GitHub write Secret, the script discovers an installation that can access the repo from url (first match). If your default branch is not main, patch manifests/demo-churn/github/cronjob.yaml env GITHUB_DEFAULT_BRANCH. The GitHub job stays Pending/Error until the hydrator write Secret exists.

Repository settings: ensure GitHub does not auto-delete *-next branches when PRs merge (Promoter relies on them). Prefer disabling Automatically delete head branches or add branch protection for a pattern such as env/** (Promoter note).

Not the GitOps Promoter webhook

This section is only for Argo CD’s Git notification endpoint (/api/webhook on the Argo CD host). GitOps Promoter uses a separate hostname (this demo: promoter-webhook.<your-domain>) and its own ingress—do not point the Argo CD Git webhook at that URL.

9. Access the Argo CD UI

Get the initial admin password:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 --decode && echo

Start a local port-forward to the Service’s HTTP port. With TLS terminated at the Ingress, the Argo CD API server runs plain HTTP on the pod; forwarding to the Service port named https (443) still reaches that HTTP listener, so a browser at https://localhost:… will try TLS and the connection will fail. Use the http port (80) and open http://, not https://.

kubectl -n argocd port-forward svc/argocd-server 8080:http
# equivalent: ... 8080:80

Then open:

http://localhost:8080

username: admin
password: value from argocd-initial-admin-secret

GitHub login (Dex)

Argo CD ships with Dex. This repository wires a GitHub OAuth2 connector in charts/argocd/values.yaml under argo-cd.configs.cm.dex.config. OAuth client ID and client secret are not in Git: they live in Secret/argocd-dex-github (keys clientId and clientSecret) with label app.kubernetes.io/part-of: argocd, delivered like other secrets via Sealed Secrets.

1. Register a GitHub OAuth App

In GitHub: Settings → Developer settings → OAuth Apps → New OAuth application.

Application name: any label you like (for example Argo CD demo)
Homepage URL: your public Argo CD URL (this demo: https://demo.<your-domain>)
Authorization callback URL: https://demo.<your-domain>/api/dex/callback (must match the url in argo-cd.configs.cm plus /api/dex/callback).

GitHub OAuth Apps allow only one authorization callback URL (unlike GitHub Apps, which allow several). You cannot register both Dex’s callback and Grafana’s …/login/github on the same OAuth app. For Grafana, use a second OAuth App whose single callback is https://grafana.<your-domain>/login/github (§10.1), or front Grafana with Dex (OIDC / generic OAuth) so GitHub still talks only to Dex.

Under the app, create a client secret.

Anonymous UI: users.anonymous.enabled is true in charts/argocd/values.yaml; unauthenticated users get policy.default (role:readonly) so the Argo CD UI is readable without signing in.

GitHub login (Dex): dex.config lists orgs / teams so only members of argoproj-labs / gitops-promoter-approvers can complete GitHub OAuth. Those users receive groups matching policy.csv and get admin. Everyone else must use anonymous readonly (or admin / local break-glass—see below). The OAuth app must still be allowed to read argoproj-labs org data or Dex cannot verify team membership—see Dex’s GitHub connector caveats.

2. Seal credentials into the repo (exact path)

From the repository root (same kubeseal controller flags as the Git webhook secret in §8):

kubectl create secret generic argocd-dex-github -n argocd \
  --from-literal=clientId='YOUR_GITHUB_OAUTH_CLIENT_ID' \
  --from-literal=clientSecret='YOUR_GITHUB_OAUTH_CLIENT_SECRET' \
  --dry-run=client -o yaml \
  | kubectl label --local --dry-run=client -f - app.kubernetes.io/part-of=argocd -o yaml \
  | kubeseal \
      --controller-name sealed-secrets \
      --controller-namespace kube-system \
      -o yaml -n argocd \
      -w charts/argocd/templates/argocd-dex-github.sealed.yaml

Commit charts/argocd/templates/argocd-dex-github.sealed.yaml and push. It is part of the Application/argocd umbrella; the SealedSecret uses sync wave -5 so it tends to apply before argo-cd subchart resources. On a brand-new cluster, Secret materialization can still lag the first API server start—if Dex errors once, wait for the controller to unseal and let Argo CD self-heal or hard refresh.

Until that Secret exists, Dex may log errors about missing client credentials; admin login (above) still works. After a successful sync, use Log in via GitHub on the Argo CD sign-in page.

3. Align org, team claims, and RBAC

dex.config uses orgs / teams plus teamNameField: slug so groups matches argoproj-labs:gitops-promoter-approvers in policy.csv. users.anonymous.enabled gives readonly without GitHub; GitHub sign-in is only for that team (admin).
If approvers still get readonly after OAuth, check org approval for the OAuth app and that policy.csv matches the actual groups claim (see Org access above).

If you fork the repo, update the g, org:team, role:admin line, orgs/teams, and your GitHub OAuth callback URL. To allow any GitHub user to sign in (not only a team), remove or widen orgs and rely on policy.default for non-admin users (Dex GitHub connector).

4. If GitHub login still fails: see DEBUGGING.md.

5. Optional: disable the local admin user

After GitHub login works, you can turn off the built-in admin account per the Argo CD FAQ (admin.enabled in argo-cd.configs.cm).

10. Create DNS records

After ingress-nginx is running, read the load balancer IPv4 address (what your A records must target):

./infra/gcp/get-ingress-lb-ip.sh

Equivalent one-liner:

kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'

If the script errors, the Service may still show <pending> under EXTERNAL-IP; wait and retry. Use kubectl -n ingress-nginx get svc for full status.

This repository assumes DNS for your domain is not in Google Cloud DNS (no managed zone is created by the GCP stack here). Create A records in your DNS provider (registrar, Cloudflare, Route 53, and so on):

demo.<your-domain>
promoter-webhook.<your-domain>
grafana.<your-domain>

Point each hostname at the ingress IP from the script. TTL around 300 seconds is reasonable while validating TLS.

10.1 Monitoring (Prometheus & Grafana)

Application/monitoring (sync wave 1, after cert-manager and ingress-nginx) installs charts/monitoring/, an umbrella over kube-prometheus-stack 82.18.0 into namespace monitoring: Prometheus Operator, Prometheus, Alertmanager, and Grafana with an Ingress on charts/monitoring/values.yaml → grafana.gitops-promoter.dev (TLS via cert-manager). The app uses syncOptions: ServerSideApply=true so large monitoring.coreos.com CRDs are not applied client-side (which would blow past the 256KiB metadata.annotations limit and fail with Too long: may not be more than 262144 bytes; requires Argo CD 2.5+).

GitOps Promoter metrics: This umbrella disables the upstream chart ServiceMonitor and ships a fixed one in charts/gitops-promoter/templates/controller-metrics-servicemonitor.yaml (see DEBUGGING.md). A Grafana dashboard lives under charts/monitoring/dashboards/gitops-promoter.json.

Grafana GitHub login: Grafana uses native GitHub OAuth (docs). Because a GitHub OAuth App only allows one callback URL (docs), Grafana needs its own OAuth App (separate from Argo’s Dex app in §9): register Homepage URL https://<your-grafana-host> and Authorization callback URL https://<your-grafana-host>/login/github (this demo: https://grafana.gitops-promoter.dev/login/github — exact URL, including https and path /login/github). Seal that app’s clientId / clientSecret into charts/monitoring/templates/grafana-github-oauth.sealed.yaml (namespace monitoring, Secret/grafana-github-oauth). Reusing the Dex app’s credentials without registering Grafana’s callback on GitHub will fail at authorize time (redirect_uri mismatch).

If GitHub shows “The redirect_uri is not associated with this application”, the client_id in the browser URL belongs to an OAuth app whose single registered callback is still something else (for example Argo’s …/api/dex/callback). Fix: create or open the Grafana OAuth app under Settings → Developer settings → OAuth Apps, set Authorization callback URL to https://grafana.gitops-promoter.dev/login/github, generate a new client secret if needed, re-seal into grafana-github-oauth, sync, and restart Grafana if it cached env vars.

kubectl create secret generic grafana-github-oauth -n monitoring \
  --from-literal=clientId='YOUR_GRAFANA_OAUTH_APP_CLIENT_ID' \
  --from-literal=clientSecret='YOUR_GRAFANA_OAUTH_APP_CLIENT_SECRET' \
  --dry-run=client -o yaml \
| kubeseal --controller-name sealed-secrets --controller-namespace kube-system -o yaml -n monitoring \
  -w charts/monitoring/templates/grafana-github-oauth.sealed.yaml

If this repository’s grafana-github-oauth.sealed.yaml was generated from Dex’s Secret, it is the wrong GitHub OAuth client for Grafana’s redirect URL—overwrite it with a seal from the Grafana-only OAuth app above.

Grafana’s deployment references Secret/grafana-github-oauth as required env vars, so the Grafana pod stays Pending until the SealedSecret is applied and unsealed (clear failure mode if the file is missing or the key does not match this cluster). Use Sign in with GitHub only: charts/monitoring/values.yaml turns off the built-in admin user (GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION) and password UI ([auth] disable_login_form, [auth.basic] enabled = false), so Helm no longer emits a chart-managed Secret whose random admin-password would change every hydration and keep Argo CD OutOfSync.

Customize Grafana hostname in charts/monitoring/values.yaml (keep ingress, tls, and grafana.ini.server.root_url consistent). To restrict sign-in to specific orgs, set auth.github.allowed_organizations again.

Prometheus is configured with serviceMonitorSelectorNilUsesHelmValues: false and an empty serviceMonitorSelector so it discovers ServiceMonitor resources in other namespaces/releases. Who can use Grafana: This demo does not set allowed_organizations, so any GitHub user may sign in. role_attribute_path assigns Editor to @argoproj-labs/gitops-promoter-approvers and Viewer to everyone else (same approvers team as Argo admin in §9). Re-login after changing roles. Security: a public URL plus open sign-in exposes metrics UIs to the whole GitHub user base—tighten with allowed_organizations, network policy, or SSO if needed.

If Argo CD stays Progressing on the Prometheus CR, see DEBUGGING.md.

After changing the umbrella chart, run helm dependency build in charts/monitoring/ (or let CI/Renovate refresh Chart.lock).

11. Verify bootstrap components

Check the Argo CD applications:

kubectl -n argocd get applications.argoproj.io

Check pods:

kubectl get pods -A

You should see these namespaces/components coming up:

argocd
cert-manager
ingress-nginx
kube-system / Sealed Secrets
monitoring (Prometheus Operator, Prometheus, Grafana)
gitops-promoter

Ingress admission webhook: the ingress-nginx chart is configured so cert-manager injects the CA into ValidatingWebhookConfiguration/ingress-nginx-admission (controller.admissionWebhooks.certManager.enabled). After sync, kubectl get validatingwebhookconfiguration ingress-nginx-admission -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | wc -c should print a non-zero length; if it stays empty, new Ingress objects can fail API validation.

12. GitOps Promoter: GitHub App, sealed credentials, and Git repository

The bootstrap install includes Application/monitoring (wave 1) so Prometheus Operator CRDs exist before the controller (Application/gitops-promoter from charts/apps, wave 3) creates its ServiceMonitor. It also includes Application/promoter-config (wave 4), which syncs promoter-config-github/ and promoter-config-gitlab/ into gitops-promoter. Until you add a real GitHub App and secret, the GitHub ScmProvider will not talk to GitHub; the GitLab path needs gitlab-scm-credentials per docs/gitlab-demo-setup.md.

Official reference: GitOps Promoter getting started (permissions, Secret shape, ScmProvider / GitRepository).

12.1 GitHub App

Create a GitHub App (org or user).
Permissions (from upstream docs): Checks read/write, Contents read/write, Pull requests read/write.

Single app for Argo + Promoter (this demo’s default). A GitHub App allows only one webhook URL. The Promoter webhook receiver does not verify X-Hub-Signature-256 today (internal/webhookreceiver/server.go in v0.25.1 / main), while Argo CD does. Until Promoter supports authenticated webhooks, point the app’s Webhook URL at Argo CD — https://demo.<your-domain>/api/webhook — with the same secret you seal into argocd-github-webhook (§8). Use that same app for ScmProvider, hydrator repository-write, and (optionally) Argo’s GitHub App webhook path in §8. GitHub will not call https://promoter-webhook.<your-domain>/ while the URL is Argo’s; Promoter still runs via the controller (polling / reconcile), only instant GitHub→Promoter delivery is deferred. When upstream adds webhook verification, split to a second app and set its webhook to promoter-webhook.<your-domain> (matching charts/gitops-promoter/values.yaml). On the app, under Subscribe to events, enable Push (Argo’s /api/webhook only handles push / ping; see §8).

Dedicated Promoter webhook (optional, second app or after split): set Webhook URL to https://promoter-webhook.<your-domain>/. Treat the receiver as an unauthenticated trigger surface until verification lands; mitigate with a non-obvious URL, network allowlists, or a validating proxy.

Webhook secret (GitHub): For the Argo-targeted URL, use a strong secret and seal it per §8. For Promoter-only URLs, GitHub may still offer a signing secret, but the controller does not validate it yet (same caveats as above).
Generate and download a private key (.pem).
Note the app’s App ID (numeric). Install the GitHub App on the user or organization that owns the repositories GitOps Promoter will use via the API (see below). Note Installation ID if you want to pin it (optional on ScmProvider; see promoter-config-github/scm-provider.yaml).

Which repositories must the installation include?

Required: every GitHub repository the Promoter mutates via the API — at minimum, whatever you name in promoter-config-github/git-repository.yaml (this demo: owner/gitops-promoter-demo — the same repo as bootstrap). Install the GitHub App on that repo (or use All repositories in a sandbox).
Argo CD clones this repo (anonymous if public); the hydrator push Secret (§8) is still required to write hydrated commits. The GitHub App on ScmProvider is what Promoter uses for API access—often the same installation as the hydrator write secret.
When you click Install, choose Only select repositories and include the repo from git-repository.yaml and any other repos your PromotionStrategy targets. All repositories is simpler for sandboxes but broader than necessary.

12.2 Repository and branches

Use the GitHub repo named in promoter-config-github/git-repository.yaml (this demo: argoproj-labs/gitops-promoter-demo — change owner / name if you fork under a different path).
promotion-strategy.yaml expects merge targets env/dev, env/e2e, and env/prd. Your hydrator / promotion flow must match the branch conventions the project documents (including env/<env>-next hydration branches). Align branch names here with what you actually create in Git.

12.3 Seal the GitHub App private key (same Application as promoter CRs)

The ScmProvider references secretRef.name: github-app-credentials. The data key must be githubAppPrivateKey (PEM string), per the getting started Secret example.

From the repository root, with the PEM file on disk (adjust paths):

kubectl create secret generic github-app-credentials -n gitops-promoter \
  --from-file=githubAppPrivateKey=/path/to/your-github-app.private-key.pem \
  --dry-run=client -o yaml \
  | kubeseal \
      --controller-name sealed-secrets \
      --controller-namespace kube-system \
      -o yaml -n gitops-promoter \
      -w charts/gitops-promoter/templates/github-app-credentials.sealed.yaml

Commit charts/gitops-promoter/templates/github-app-credentials.sealed.yaml. The gitops-promoter Application applies it alongside the gitops-promoter Helm release (sync wave -5 on the SealedSecret).

12.4 Wire non-secret IDs in Git

Edit promoter-config-github/scm-provider.yaml: set spec.github.appID to your numeric App ID. Uncomment and set installationID only if you need to pin a single installation.

Edit promoter-config-github/git-repository.yaml if owner / name differ from the GitHub repository that holds demo-apps/guestbook and the env/... promotion branches (normally your gitops-promoter-demo fork).

Push; after sync, kubectl -n gitops-promoter get sealedsecret,secret,scmprovider should show the decoded Secret and a healthy ScmProvider.

12.5 Argo CD UI and other secrets

Grafana dashboards and Prometheus: charts/monitoring/ and §10.1.
Argo CD integrations (extension, links, commit-status keys): charts/argocd/values.yaml and GitOps Promoter Argo CD integrations. If a PromotionStrategy is not top-level in an Application’s resource tree, set promoter.argoproj.io/has-promotionstrategy: "true" on that Application so the extension tab appears.
Argo CD Git webhook HMAC: §8 → charts/argocd/templates/argocd-github-webhook.sealed.yaml.
Argo CD Dex GitHub OAuth: §9 → charts/argocd/templates/argocd-dex-github.sealed.yaml.
Hydrator repository-write: §8 → charts/argocd/templates/argocd-repo-gitops-promoter-write.sealed.yaml.
GitLab guestbook hydrator repository-write and gitlab-demo-churn-credentials: docs/gitlab-demo-setup.md → charts/argocd/templates/argocd-repo-gitlab-guestbook-write.sealed.yaml, gitlab-demo-churn-credentials.sealed.yaml.

12.6 Checklist (other prerequisites)

Create GitHub App, install it with access to the repo in git-repository.yaml (see §12.1). By default point the app webhook at Argo CD (https://demo.<your-domain>/api/webhook); use a second app with promoter-webhook.<your-domain> only when you want a dedicated Promoter webhook (for example after Promoter verifies signatures).
Ensure branches in that repo match promotion-strategy.yaml (and your hydrator).
Seal and commit charts/gitops-promoter/templates/github-app-credentials.sealed.yaml.
Set appID (and optionally installationID) in promoter-config-github/scm-provider.yaml; fix git-repository.yaml owner/name.
Optional: §8 / §9 sealed files for Argo CD.

13. Suggested first commits

A clean sequence is:

Bootstrap commit
- apps/root-app.yaml and charts/apps/ (Helm chart of child **Application**s; Argo / Promoter sealed files can land in follow-up commits under charts/argocd/templates/ and charts/gitops-promoter/templates/)
- charts/ (other umbrella charts)
- manifests/ (excluding demo-churn/**; churn syncs via Application/demo-churn)
Promoter config + secrets commit
- promoter-config-github/, promoter-config-gitlab/ (CRs; charts/gitops-promoter/templates/github-app-credentials.sealed.yaml when the GitHub App exists; GitLab token gitlab-scm-credentials per docs/gitlab-demo-setup.md)
- other sealed secrets as needed under charts/argocd/templates/ (webhook, Dex, GitHub hydrator write, GitLab guestbook write + churn per docs/gitlab-demo-setup.md)
Demo workloads commit
- demo-apps/guestbook/ and guestbook Application templates under charts/apps/templates/
Monitoring commit
- charts/monitoring/ and the Application/monitoring template in charts/apps (run helm dependency build; grafana-github-oauth.sealed.yaml is tied to this cluster’s Sealed Secrets key)

This keeps the cluster bring-up simple and avoids introducing broken credentials too early.

Version pins currently used

Bootstrap charts are currently pinned to:

Argo CD 9.4.17
cert-manager 1.20.1
GitOps Promoter 0.5.1 (Helm chart; Argo CD UI extension bundle v0.25.1 per GitOps Promoter Argo CD integrations)
ingress-nginx 4.15.1
kube-prometheus-stack 82.18.0 (charts/monitoring/ umbrella)
sealed-secrets 2.18.4

Update the dependency versions in the umbrella chart Chart.yaml files when you upgrade.

Notes

infra/gcp/terraform/terraform.tfvars holds OpenTofu inputs for this demo; replace values when you fork or reuse the stack elsewhere.
.terraform.lock.hcl is safe to commit and helps keep provider resolution reproducible (regenerate with tofu init or terraform init after provider bumps).
The root app must point at a repository/branch that already contains the committed bootstrap manifests.
Transient Unknown sync status or other oddities: DEBUGGING.md.

Uh oh!

FilesExpand file tree

SETUP.md

Latest commit

History