HYPERFLEET-937 - feat: add initSidecars support for Kubernetes native sidecar containers

Adds initSidecars list for injecting init containers with restartPolicy: Always
(Kubernetes 1.28+). These start before db-migrate, solving the deadlock where
database proxies in regular sidecars aren't available during migrations.

Author: rafabene

Makefile

@@ -348,6 +348,27 @@ test-helm: ## Test Helm charts (lint, template, validate)
 		--set-json 'sidecars=[{"name":"test-sidecar","image":"busybox:1.36","command":["sleep","infinity"]}]' > /dev/null
 	@echo "Sidecar injection config template OK"
 	@echo ""
+	@echo "Testing template with native init sidecar injection..."
+	helm template test-release charts/ \
+		--set image.registry=quay.io \
+		--set image.repository=openshift-hyperfleet/hyperfleet-api \
+		--set image.tag=test \
+		--set 'adapters.cluster=["validation"]' \
+		--set 'adapters.nodepool=["validation"]' \
+		--set-json 'initSidecars=[{"name":"cloud-sql-proxy","restartPolicy":"Always","image":"gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.14.3","args":["--structured-logs","--port=5432","project:region:instance"]}]' > /dev/null
+	@echo "Init sidecar injection config template OK"
+	@echo ""
+	@echo "Testing template with init sidecars and no database..."
+	helm template test-release charts/ \
+		--set image.registry=quay.io \
+		--set image.repository=openshift-hyperfleet/hyperfleet-api \
+		--set image.tag=test \
+		--set 'adapters.cluster=["validation"]' \
+		--set 'adapters.nodepool=["validation"]' \
+		--set database.postgresql.enabled=false \
+		--set-json 'initSidecars=[{"name":"test-proxy","restartPolicy":"Always","image":"busybox:1.36","command":["sleep","infinity"]}]' > /dev/null
+	@echo "Init sidecar without database config template OK"
+	@echo ""
 	@echo "Testing template with full adapter config..."
 	helm template test-release charts/ \
 		--set image.registry=quay.io \

charts/templates/deployment.yaml

@@ -39,8 +39,11 @@ spec:
       serviceAccountName: {{ include "hyperfleet-api.serviceAccountName" . }}
       securityContext:
         {{- toYaml .Values.podSecurityContext | nindent 8 }}
-      {{- if or .Values.database.postgresql.enabled .Values.database.external.enabled }}
+      {{- if or .Values.initSidecars (or .Values.database.postgresql.enabled .Values.database.external.enabled) }}
       initContainers:
+      {{- range .Values.initSidecars }}
+      - {{- toYaml . | nindent 8 }}
+      {{- end }}
       {{- if .Values.database.postgresql.enabled }}
       - name: wait-for-db
         image: busybox:1.36

charts/values.yaml

@@ -278,10 +278,44 @@ tracing:
   samplerArg: "1.0"
   propagators: "tracecontext,baggage"
 
+# ============================================================
+# Init Sidecar Containers (Kubernetes Native Sidecars)
+# ============================================================
+# Native sidecars are init containers with restartPolicy: Always (Kubernetes 1.28+).
+# They start BEFORE other init containers and keep running throughout the pod lifecycle.
+# Use this for database proxies that must be available during db-migrate.
+# Each entry is a full Kubernetes container spec — you MUST include restartPolicy: Always.
+initSidecars: []
+# Example: Cloud SQL Auth Proxy as native sidecar
+#   - name: cloud-sql-proxy
+#     restartPolicy: Always
+#     image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.14.3
+#     args:
+#       - "--structured-logs"
+#       - "--port=5432"
+#       - "PROJECT:REGION:INSTANCE"
+#     securityContext:
+#       allowPrivilegeEscalation: false
+#       capabilities:
+#         drop: [ALL]
+#       readOnlyRootFilesystem: true
+#       runAsNonRoot: true
+#       seccompProfile:
+#         type: RuntimeDefault
+#     resources:
+#       requests:
+#         cpu: 100m
+#         memory: 64Mi
+#       limits:
+#         cpu: 200m
+#         memory: 128Mi
+
 # ============================================================
 # Sidecar Containers
 # ============================================================
-# Generic sidecar injection — each entry is a full Kubernetes container spec.
+# Regular sidecar injection — each entry is a full Kubernetes container spec.
+# These start AFTER init containers complete. Use initSidecars above for
+# containers that must be available during init (e.g., database proxies).
 # Use extraVolumes/extraVolumeMounts for sidecar volume needs.
 # Operators SHOULD enforce container security via admission policies (PodSecurity Standards, Kyverno, etc.).
 sidecars: []
@@ -324,20 +358,6 @@ sidecars: []
 #       requests:
 #         cpu: 50m
 #         memory: 64Mi
-#
-# Example: Cloud SQL Auth Proxy (GCP)
-#   - name: cloud-sql-proxy
-#     image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.14.3
-#     args:
-#       - "--structured-logs"
-#       - "--port=5432"
-#       - "PROJECT:REGION:INSTANCE"
-#     securityContext:
-#       runAsNonRoot: true
-#     resources:
-#       requests:
-#         cpu: 100m
-#         memory: 64Mi
 
 # ============================================================
 # Advanced Overrides (Escape Hatch)

docs/database.md

@@ -184,11 +184,55 @@ The readiness probe (`/readyz`) pings the database with a separate timeout contr
 
 ## Sidecar Containers
 
-The Helm chart supports generic sidecar injection via the `sidecars` list in `values.yaml`. Each entry is a full Kubernetes container spec injected into the deployment pod. This can be used for any purpose — database proxies, log shippers, monitoring agents, etc.
+The Helm chart supports two sidecar injection mechanisms:
 
-A common use case is database proxy sidecars (PgBouncer, Cloud SQL Auth Proxy). The example below shows a PgBouncer configuration.
+| List | Where it renders | Starts when | Use case |
+|------|------------------|-------------|----------|
+| `initSidecars` | `initContainers` (with `restartPolicy: Always`) | Before all other init containers | Database proxies that must be available during `db-migrate` |
+| `sidecars` | `containers` | After all init containers complete | Log shippers, monitoring agents, connection poolers that don't need to run during init |
 
-### Example: PgBouncer Sidecar
+Each entry in either list is a full Kubernetes container spec injected as-is into the deployment pod.
+
+### Native Sidecars (`initSidecars`)
+
+Kubernetes 1.28+ supports [native sidecar containers](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) — init containers declared with `restartPolicy: Always`. They start before other init containers and keep running throughout the pod lifecycle.
+
+Use `initSidecars` for database proxies (Cloud SQL Auth Proxy, AlloyDB Auth Proxy) that must be reachable when the `db-migrate` init container runs. Without this, the migration deadlocks: the init container waits for the proxy, but regular sidecars only start after all init containers finish.
+
+### Example: Cloud SQL Auth Proxy (Native Sidecar)
+
+```yaml
+# values.yaml
+initSidecars:
+  - name: cloud-sql-proxy
+    restartPolicy: Always
+    image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.14.3
+    args:
+      - "--structured-logs"
+      - "--port=5432"
+      - "PROJECT:REGION:INSTANCE"
+    securityContext:
+      allowPrivilegeEscalation: false
+      capabilities:
+        drop: [ALL]
+      readOnlyRootFilesystem: true
+      runAsNonRoot: true
+      seccompProfile:
+        type: RuntimeDefault
+    resources:
+      requests:
+        cpu: 100m
+        memory: 64Mi
+      limits:
+        cpu: 200m
+        memory: 128Mi
+```
+
+With this setup, both the `db-migrate` init container and the runtime API container connect through the proxy — no `extraEnv` override needed since the proxy listens on `localhost:5432`.
+
+### Regular Sidecars (`sidecars`)
+
+Use the `sidecars` list for containers that don't need to be available during init. The example below shows a PgBouncer connection pooler:
 
 ```yaml
 # values.yaml
@@ -233,7 +277,7 @@ sidecars:
         memory: 64Mi
 ```
 
-When using a proxy sidecar, the API container must connect to the proxy instead of PostgreSQL directly. The `database.external.secretName` secret must contain the **direct** database endpoint (host/port) so the `db-migrate` init container can run migrations before sidecars start. To route runtime traffic through the proxy, use `extraEnv` overrides in the main container:
+When using a regular sidecar proxy, the `db-migrate` init container connects directly to the database (regular sidecars aren't running yet). Route runtime traffic through the proxy with `extraEnv`:
 
 ```yaml
 extraEnv:
@@ -245,29 +289,33 @@ extraEnv:
 
 Use `extraVolumes` and `extraVolumeMounts` for any volumes the sidecar requires (e.g., temp dirs, config dirs).
 
-### Proxy Architecture
+### Architecture
 
 ```text
-┌─────────────────────────────────────────┐
-│  Pod                                    │
-│                                         │
-│  ┌──────────────┐     ┌──────────┐      │
-│  │  hyperfleet   │────▶│  proxy   │─────┼──▶ Database
-│  │  API          │     │ sidecar  │      │
-│  │  (localhost)  │     └──────────┘      │
-│  └──────────────┘                       │
-│                                         │
-│  Init containers (migrate) ─────────────┼──▶ Database
-│  (direct connection, bypasses proxy)     │
-└─────────────────────────────────────────┘
+┌────────────────────────────────────────────────────────┐
+│  Pod                                                   │
+│                                                        │
+│  initSidecars (restartPolicy: Always)                  │
+│  ┌──────────────────┐                                  │
+│  │  cloud-sql-proxy  │◄─── starts first, stays running │
+│  └────────┬─────────┘                                  │
+│           │                                            │
+│  initContainers                                        │
+│  ┌──────────────────┐                                  │
+│  │  db-migrate       │──▶ proxy ──▶ Database           │
+│  └──────────────────┘                                  │
+│                                                        │
+│  containers                                            │
+│  ┌──────────────────┐                                  │
+│  │  hyperfleet-api   │──▶ proxy ──▶ Database           │
+│  └──────────────────┘                                  │
+└────────────────────────────────────────────────────────┘
 ```
 
-- Init containers (migrations) should connect directly to the database — they run before sidecars start, and DDL operations may not be compatible with connection pooling.
-
 ### Common Proxy Choices
 
-- **PgBouncer**: Lightweight connection pooler. Use `transaction` pool mode for stateless APIs. See commented example in `values.yaml`.
-- **Cloud SQL Auth Proxy**: Required for GCP Cloud SQL access without complex network/IP setup. See commented example in `values.yaml`.
+- **Cloud SQL Auth Proxy**: Required for GCP Cloud SQL. Use `initSidecars` so migrations can reach the database through the proxy.
+- **PgBouncer**: Lightweight connection pooler. Use `transaction` pool mode for stateless APIs. Can go in either `initSidecars` or `sidecars` depending on whether migrations need it.
 
 ## Related Documentation