Skip to content

KUSTOMIZE_INSTANCE_TEMPLATE.md

Latest commit

 

History

History
115 lines (83 loc) · 7.7 KB

KUSTOMIZE_INSTANCE_TEMPLATE.md

File metadata and controls

115 lines (83 loc) · 7.7 KB
Raw

Kustomize Instance Template

Overview

The instance template is a parameterized Kustomize overlay at deployments/openshift/kustomize/overlays/instance-template/ that generates fully isolated, instance-specific Kubernetes manifests. It enables multiple independent deployments of the full application stack within a single OpenShift namespace.

How It Works

The template uses placeholder tokens that the deploy script replaces with actual values before running kustomize build. This generates a complete set of Kubernetes resources where every resource is prefixed, labeled, and configured for a specific instance.

Placeholder Tokens

Token Description Example Value
__INSTANCE_NAME__ Sanitized instance name (max 20 chars) feature-deployment-f
__NAMESPACE__ OpenShift namespace fd34fb-prod
__CLUSTER_DOMAIN__ Cluster wildcard domain apps.silver.devops.gov.bc.ca
__BACKEND_IMAGE__ Backend container image (without tag) <artifactory-url>/kfd3-fd34fb-local/backend-services
__FRONTEND_IMAGE__ Frontend container image (without tag) <artifactory-url>/kfd3-fd34fb-local/frontend
__WORKER_IMAGE__ Temporal worker container image (without tag) <artifactory-url>/kfd3-fd34fb-local/temporal
__IMAGE_TAG__ Image tag for all services (max 128 chars, decoupled from instance name) feature-deployment-features
__SSO_AUTH_SERVER_URL__ Keycloak/SSO authentication server URL https://sso.example.com/auth
__SSO_REALM__ SSO realm name my-realm
__SSO_CLIENT_ID__ SSO client identifier my-client

Kustomize Features Used

  • namePrefix: Prefixes all resource names with <instance-name>-. Kustomize automatically updates cross-references (Service selectors, ConfigMap/Secret refs, PVC claims).
  • commonLabels: Adds app.kubernetes.io/instance: <instance-name> to all resources, including pod template labels and selector matchLabels.
  • images: Overrides base image references to point to Artifactory images for the current branch.
  • patches: Updates hardcoded in-cluster service references, sets route hostnames, fixes operator-managed secret references, and configures SSO settings.
  • configurations: References kustomize-config.yml which defines nameReference rules so Kustomize auto-updates Route spec.to.name when namePrefix is applied.

Route IP restrictions

The source-IP allowlist is production-only. Base Route manifests under deployments/openshift/kustomize/base/ carry no allowlist, so ephemeral PR and test/dev instances are reachable without VPN. For production namespaces the components/prod-resources component adds haproxy.router.openshift.io/ip_allowlist to both public Routes, so the OpenShift router (HAProxy) only allows traffic whose source IP is in the BC Gov public 142.x allocation (ARIN org PBC-51-Z): 142.22.0.0/16–142.36.0.0/16, space-separated. This covers all BC Gov networks including VPN egress and the Silver/Gold/Gold-DR NAT pools (all within 142.34.0.0/16); other clients receive HTTP 403 at the router.

scripts/lib/generate-overlay.sh includes components/prod-resources automatically when the target namespace ends in -prod — the same mechanism used for prod-only memory limits and PVC sizes. There is no per-overlay allowlist patch; the legacy overlays/{prod,test,dev} directories are not part of the deploy path (the Deploy Instance workflow always renders from overlays/instance-template).

The backend Route keeps haproxy.router.openshift.io/deny-list for /metrics in base, so /metrics is blocked in every environment (see docs-md/PROMETHEUS_METRICS.md).

Annotation key: ip_allowlist (renamed from ip_whitelist in PR #218 / AI-1341 for inclusive naming).

History: a prior base value of 142.16.0.0/11 both over-restricted (a /11 only spans 142.0.0.0–142.31.255.255, silently excluding BC Gov clients in 142.32.0.0/16–142.36.0.0/16) and applied to every environment, including test/dev.

What Gets Patched

Kustomize's namePrefix handles most cross-references automatically, but several categories of references require explicit patches:

Hardcoded string values (env vars / ConfigMap data)

  • Temporal server deployment: POSTGRES_SEEDS env var updated to reference the prefixed PostgreSQL service (temporal-pg-primary)
  • Temporal UI deployment: TEMPORAL_ADDRESS env var updated to reference the prefixed Temporal service
  • Backend services ConfigMap: TEMPORAL_ADDRESS, FRONTEND_URL, BACKEND_URL, SSO_REDIRECT_URI, and SSO settings (SSO_AUTH_SERVER_URL, SSO_REALM, SSO_CLIENT_ID)
  • Temporal worker ConfigMap: TEMPORAL_ADDRESS updated to reference the prefixed Temporal service
  • Route hostnames: Set to <instance>-<service>-<namespace>.<cluster-domain> (single level under wildcard cert to avoid ERR_CERT_COMMON_NAME_INVALID)
  • NetworkPolicies: Scoped to only allow ingress from pods with the same instance label and from the OpenShift ingress router

Operator-managed secret references (not auto-prefixed by Kustomize)

Crunchy PostgreSQL operator creates secrets with names derived from the PostgresCluster resource name. Since Kustomize doesn't manage these secrets (they're created by the operator at runtime), secretKeyRef names in deployments must be patched explicitly:

  • Backend services: migrate-db init container and backend-services container both reference <instance>-app-pg-pguser-admin
  • Temporal server: config-init and schema-setup init containers plus temporal container reference <instance>-temporal-pg-pguser-temporal (6 total secretKeyRef entries)
  • Temporal worker: References <instance>-app-pg-pguser-admin

PostgresCluster databaseInitSQL

  • Temporal PostgresCluster: databaseInitSQL.name patched to <instance>-temporal-postgres-init-sql (Kustomize doesn't auto-update this CRD field)

Instance Isolation

Each instance gets its own complete, independent stack:

  • Crunchy PostgreSQL cluster (app database)
  • Crunchy PostgreSQL cluster (Temporal database)
  • Temporal server, worker, and UI (UI is not publicly exposed — use oc port-forward for local access)
  • Backend services deployment
  • Frontend deployment
  • Routes for frontend and backend (with instance-specific hostnames)
  • ConfigMaps and Secrets
  • PVCs
  • NetworkPolicies (scoped to instance label, preventing cross-instance traffic)

Overlay Generation Library

The scripts/lib/generate-overlay.sh library provides functions to generate and clean up instance-specific overlays:

source scripts/lib/generate-overlay.sh

# Generate an overlay
OVERLAY_DIR=$(generate_instance_overlay \
  --instance "feature-my-thing" \
  --namespace "fd34fb-prod" \
  --cluster-domain "apps.silver.devops.gov.bc.ca" \
  --backend-image "<artifactory-url>/kfd3-fd34fb-local/backend-services" \
  --frontend-image "<artifactory-url>/kfd3-fd34fb-local/frontend" \
  --worker-image "<artifactory-url>/kfd3-fd34fb-local/temporal" \
  --image-tag "feature-my-thing")

# Use the overlay
oc apply -k "${OVERLAY_DIR}"

# Clean up
cleanup_generated_overlay "${OVERLAY_DIR}"

The function copies the template to a nested temporary directory structure (tmpdir/overlays/instance/) with a symlink (tmpdir/base -> real base dir) so the relative path ../../base in the kustomization resolves correctly. It replaces all placeholder tokens and returns the path. The caller is responsible for cleanup via cleanup_generated_overlay.

Testing

bash scripts/lib/generate-overlay.test.sh

Existing Overlays

The instance template is additive. Existing overlays at deployments/openshift/kustomize/overlays/ (dev/, test/, prod/) are not modified.