Pattern A: Flux HelmRelease GitOps migration (closes GH013) (#1089)

Closes #1088.

WHY
---
The deploy-azd.yml `commit-rendered-manifests` job pushed bot-generated
manifests directly to refs/heads/main. The `main-governance-baseline`
ruleset (id 14638366) rejects this push with GH013, blocking every
deploy after PRs merge. Bypass-actor and orphan-branch alternatives
were rejected as anti-patterns.

WHAT
----
Roll out Pattern A (Flux HelmRelease + in-cluster Helm rendering) to
all 27 AKS services. The helm-controller renders the chart in-cluster
on every reconciliation, so no rendered YAML lives in git and no
workflow ever pushes back to main.

CHANGES
-------
- 24 new HelmRelease YAMLs in .kubernetes/releases/agents/ + 1 in
  .kubernetes/releases/crud/. Generator preserves every env var,
  resource limit, AGC route, command/args override, and UAMI binding
  from the previously deployed cluster state. Skips the 3 already-
  migrated services (ecommerce-catalog-search, truth-enrichment,
  truth-hitl).
- .kubernetes/releases/agents/kustomization.yaml updated to list all
  26 agent HelmReleases.
- .kubernetes/releases/crud/kustomization.yaml created listing the
  crud-service HelmRelease.
- Bicep fluxConfig switched from `.kubernetes/rendered/{crud,agents}`
  to `.kubernetes/releases/{crud,agents}`. CRUD reconciles first;
  agents depend on it.
- deploy-azd.yml: removed `commit-rendered-manifests` job and rewired
  `wait-flux-reconciliation` to depend on deploy-crud / deploy-agents
  directly. Renamed misleading "for Flux commit" artifact step labels
  to "for verification" since these artifacts now feed only the
  prompt-verification flow.
- ADR-017 amended with Phase 2 completion notes and Phase 2b plan
  (Flux ImageUpdateAutomation with PR bridge for image tag updates).

VALIDATION
----------
- All 27 HelmReleases pass `helm template` against
  .kubernetes/chart (validate_helmreleases.py)
- `kubectl kustomize .kubernetes/releases/agents` resolves 26
  HelmReleases; `.kubernetes/releases/crud` resolves 1
- scripts/ci/validate_k8s_name_length.py: passes
- Workflow YAML parses cleanly (yaml.safe_load)

KNOWN GAP (Phase 2b, separate epic)
------------------------------------
After merge, `azd deploy` continues to kubectl-apply new image tags.
Within 5 min Flux can revert to the older tag committed in the
HelmRelease YAML. HelmRelease tags here reflect the currently-deployed
images at PR time, so the first reconciliation after Bicep redeploy
is a no-op. Phase 2b closes the loop with Flux
ImageRepository/ImagePolicy/ImageUpdateAutomation and a PR-bridge for
protected branches.

Author: Cataldir

.github/workflows/deploy-azd.yml

@@ -2057,7 +2057,7 @@ jobs:
           POSTGRES_USER: ${{ needs.provision.outputs.POSTGRES_AUTH_MODE == 'password' && needs.provision.outputs.POSTGRES_ADMIN_USER || needs.provision.outputs.POSTGRES_USER }}
           POSTGRES_DATABASE: ${{ needs.provision.outputs.POSTGRES_DATABASE }}
 
-      - name: Upload rendered CRUD manifest for Flux commit
+      - name: Upload rendered CRUD manifest for verification
         uses: actions/upload-artifact@v4
         with:
           name: rendered-manifest-crud-service
@@ -2710,91 +2710,29 @@ jobs:
           POSTGRES_USER: ${{ needs.provision.outputs.POSTGRES_AUTH_MODE == 'password' && needs.provision.outputs.POSTGRES_ADMIN_USER || needs.provision.outputs.POSTGRES_USER }}
           POSTGRES_DATABASE: ${{ needs.provision.outputs.POSTGRES_DATABASE }}
 
-      - name: Upload rendered agent manifest for Flux commit
+      - name: Upload rendered agent manifest for verification
         if: ${{ success() }}
         uses: actions/upload-artifact@v4
         with:
           name: rendered-manifest-${{ matrix.service }}
           path: .kubernetes/rendered/${{ matrix.service }}/all.yaml
           retention-days: 1
 
-  commit-rendered-manifests:
+  # NOTE (ADR-017 amendment, Pattern A): the previous `commit-rendered-manifests`
+  # job was deleted because it pushed bot-generated manifests directly to
+  # `refs/heads/main`, which is rejected by the `main-governance-baseline`
+  # ruleset (GH013). Flux now reconciles HelmRelease CRDs from
+  # `.kubernetes/releases/{crud,agents}` and the helm-controller renders the
+  # chart in-cluster on every reconciliation, so no workflow push to main is
+  # ever required. Image tag updates remain a follow-up (planned: Flux
+  # ImageUpdateAutomation with PR bridge).
+
+  wait-flux-reconciliation:
     runs-on: ubuntu-latest
     if: ${{ always() && !cancelled() && !inputs.uiOnly && (needs.deploy-crud.result == 'success' || needs.deploy-agents.result == 'success') }}
     needs:
-      - detect-changes
-      - provision
       - deploy-crud
       - deploy-agents
-      - build-aks-images
-    permissions:
-      contents: write
-    steps:
-      - name: Resolve publication branch
-        id: publication-branch
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          SOURCE_REF="${DEPLOY_SOURCE_REF}"
-
-          if [[ "$SOURCE_REF" != refs/heads/* ]]; then
-            echo "::error::Rendered manifest publication requires a pushable branch ref, but DEPLOY_SOURCE_REF='$SOURCE_REF'."
-            exit 1
-          fi
-
-          BRANCH_NAME="${SOURCE_REF#refs/heads/}"
-          if [ -z "$BRANCH_NAME" ]; then
-            echo "::error::Rendered manifest publication could not resolve a branch name from DEPLOY_SOURCE_REF='$SOURCE_REF'."
-            exit 1
-          fi
-
-          echo "branch_ref=$SOURCE_REF" >> "$GITHUB_OUTPUT"
-          echo "branch_name=$BRANCH_NAME" >> "$GITHUB_OUTPUT"
-
-      - uses: actions/checkout@v4
-        with:
-          ref: ${{ steps.publication-branch.outputs.branch_ref }}
-          fetch-depth: 1
-          token: ${{ github.token }}
-
-      - name: Download rendered manifests
-        uses: actions/download-artifact@v4
-        with:
-          pattern: rendered-manifest-*
-          path: ${{ runner.temp }}/rendered-artifacts
-
-      - name: Copy rendered manifests into repo
-        shell: bash
-        run: |
-          set -euo pipefail
-          for artifact_dir in "${RUNNER_TEMP}"/rendered-artifacts/rendered-manifest-*; do
-            [ -d "$artifact_dir" ] || continue
-            svc_name=$(basename "$artifact_dir" | sed 's/^rendered-manifest-//')
-            mkdir -p ".kubernetes/rendered/${svc_name}"
-            cp "${artifact_dir}/all.yaml" ".kubernetes/rendered/${svc_name}/all.yaml"
-            echo "Copied rendered manifest for ${svc_name}"
-          done
-
-      - name: Commit rendered manifests for Flux reconciliation
-        shell: bash
-        run: |
-          set -euo pipefail
-          git config user.name "github-actions[bot]"
-          git config user.email "github-actions[bot]@users.noreply.github.com"
-          git add .kubernetes/rendered/ || true
-          if git diff --cached --quiet; then
-            echo "No rendered manifest changes to commit."
-            exit 0
-          fi
-          git commit -m "deploy: update rendered manifests [skip ci]"
-          git push origin "HEAD:refs/heads/${{ steps.publication-branch.outputs.branch_name }}"
-
-  wait-flux-reconciliation:
-    runs-on: ubuntu-latest
-    if: ${{ !inputs.uiOnly && (needs.commit-rendered-manifests.result == 'success') }}
-    needs:
-      - commit-rendered-manifests
       - detect-changes
     environment: ${{ inputs.githubEnvironment }}
     env:
@@ -4203,7 +4141,6 @@ jobs:
       - detect-changes
       - deploy-crud
       - deploy-agents
-      - commit-rendered-manifests
       - wait-flux-reconciliation
       - sync-apim
       - sync-apic
@@ -4299,7 +4236,6 @@ jobs:
     runs-on: ubuntu-latest
     if: ${{ always() && !cancelled() && !inputs.uiOnly }}
     needs:
-      - commit-rendered-manifests
       - wait-flux-reconciliation
       - validate-agc-readiness
       - sync-apim

.infra/modules/shared-infrastructure/shared-infrastructure.bicep

@@ -1218,7 +1218,9 @@ resource fluxExtension 'Microsoft.KubernetesConfiguration/extensions@2023-05-01'
   }
 }
 
-// Flux GitOps configuration — reconciles rendered manifests from the repository.
+// Flux GitOps configuration — reconciles HelmRelease CRDs from the repository.
+// Pattern A (ADR-017 amended): the helm-controller renders the chart in-cluster on
+// every reconciliation, so no workflow ever pushes rendered YAML back to main.
 resource fluxConfig 'Microsoft.KubernetesConfiguration/fluxConfigurations@2024-04-01-preview' = {
   name: 'holiday-peak-gitops'
   scope: aksClusterResource
@@ -1239,15 +1241,15 @@ resource fluxConfig 'Microsoft.KubernetesConfiguration/fluxConfigurations@2024-0
     }
     kustomizations: {
       'holiday-peak-crud': {
-        path: '.kubernetes/rendered/crud'
+        path: '.kubernetes/releases/crud'
         syncIntervalInSeconds: 300
         timeoutInSeconds: 600
         prune: true
         force: false
         dependsOn: []
       }
       'holiday-peak-agents': {
-        path: '.kubernetes/rendered/agents'
+        path: '.kubernetes/releases/agents'
         syncIntervalInSeconds: 300
         timeoutInSeconds: 600
         prune: true

.kubernetes/releases/agents/crm-campaign-intelligence.yaml

@@ -0,0 +1,105 @@
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: crm-campaign-intelligence
+  namespace: flux-system
+spec:
+  targetNamespace: holiday-peak-agents
+  releaseName: crm-campaign-intelligence
+  interval: 5m
+  timeout: 10m
+  chart:
+    spec:
+      chart: .kubernetes/chart
+      sourceRef:
+        kind: GitRepository
+        name: holiday-peak-gitops
+        namespace: flux-system
+      interval: 5m
+  upgrade:
+    remediation:
+      retries: 3
+  install:
+    createNamespace: false
+    remediation:
+      retries: 3
+  values:
+    serviceName: crm-campaign-intelligence
+    serviceAccount:
+      create: true
+      clientId: 036f4fd2-9093-4948-b0dd-beb5c87aa6dc
+    image:
+      repository: holidaypeakhub405devacr.azurecr.io/holiday-peak-hub/crm-campaign-intelligence-dev
+      tag: azd-deploy-1775344528
+    replicaCount: 2
+    resources:
+      limits:
+        cpu: 500m
+        memory: 512Mi
+      requests:
+        cpu: 250m
+        memory: 256Mi
+    nodeSelector:
+      agentpool: agents
+    tolerations:
+    - effect: NoSchedule
+      key: workload
+      operator: Equal
+      value: agents
+    availability:
+      strategy:
+        type: RollingUpdate
+      rollingUpdate:
+        maxUnavailable: 25%
+        maxSurge: 25%
+    pdb:
+      enabled: false
+    keda:
+      enabled: false
+    agc:
+      enabled: true
+      gatewayClassName: azure-alb-external
+      hostnames:
+      - esbcc8bcfyazbbdg.fz03.alb.azure.com
+      parentRefs:
+      - name: holiday-peak-agc
+        namespace: holiday-peak-crud
+      paths:
+      - path: /crm-campaign-intelligence
+        pathType: PathPrefix
+        rewriteTo: /
+    env:
+      AI_SEARCH_AUTH_MODE: managed_identity
+      AI_SEARCH_ENDPOINT: https://holidaypeakhub405devsearch.search.windows.net
+      AI_SEARCH_INDEX: catalog-products
+      AI_SEARCH_INDEXER_NAME: search-enriched-products-indexer
+      AI_SEARCH_VECTOR_INDEX: product_search_index
+      APPLICATIONINSIGHTS_CONNECTION_STRING: InstrumentationKey=d2fb45bc-e648-4da2-9345-d278636aede5;IngestionEndpoint=https://centralus-2.in.applicationinsights.azure.com/;LiveEndpoint=https://centralus.livediagnostics.monitor.azure.com/;ApplicationId=d8eb64c4-956d-46ab-a02e-d481deadaa0b
+      AZURE_CLIENT_ID: 036f4fd2-9093-4948-b0dd-beb5c87aa6dc
+      AZURE_TENANT_ID: 16b3c013-d300-468d-ac64-7eda0820b6d3
+      BLOB_ACCOUNT_URL: https://holidaypeakhub405devstor.blob.core.windows.net
+      BLOB_CONTAINER: agent-memory
+      CATALOG_SEARCH_REQUIRE_AI_SEARCH: 'true'
+      COSMOS_ACCOUNT_URI: https://holidaypeakhub405-dev-cosmos.documents.azure.com:443/
+      COSMOS_CONTAINER: agent-memory
+      COSMOS_DATABASE: holiday-peak-db
+      EMBEDDING_DEPLOYMENT_NAME: text-embedding-3-large
+      EVENT_HUB_NAMESPACE: holidaypeakhub405-dev-eventhub
+      FOUNDRY_AGENT_NAME_FAST: crm-campaign-intelligence-fast
+      FOUNDRY_AGENT_NAME_RICH: crm-campaign-intelligence-rich
+      FOUNDRY_AUTO_ENSURE_ON_STARTUP: 'true'
+      FOUNDRY_STREAM: 'true'
+      AGENT_FOUNDRY_INVOKE_TIMEOUT_SECONDS: '60'
+      FOUNDRY_STRICT_ENFORCEMENT: 'false'
+      KEY_VAULT_URI: https://holidaypeakhub405-dev-kv.vault.azure.net/
+      MODEL_DEPLOYMENT_NAME_FAST: gpt-5-nano
+      MODEL_DEPLOYMENT_NAME_RICH: gpt-5
+      POSTGRES_AUTH_MODE: entra
+      POSTGRES_DATABASE: holiday_peak_crud
+      POSTGRES_HOST: holidaypeakhub405-dev-postgres.postgres.database.azure.com
+      POSTGRES_SSL: 'true'
+      POSTGRES_USER: holidaypeakhub405-dev-crud-identity
+      PROJECT_ENDPOINT: https://holidaypeakhub405devais.services.ai.azure.com/api/projects/aipholidaris
+      PROJECT_NAME: aipholidaris
+      REDIS_HOST: holidaypeakhub405-dev-redis.redis.cache.windows.net
+      REDIS_PASSWORD_SECRET_NAME: redis-primary-key

.kubernetes/releases/agents/crm-profile-aggregation.yaml

@@ -0,0 +1,105 @@
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: crm-profile-aggregation
+  namespace: flux-system
+spec:
+  targetNamespace: holiday-peak-agents
+  releaseName: crm-profile-aggregation
+  interval: 5m
+  timeout: 10m
+  chart:
+    spec:
+      chart: .kubernetes/chart
+      sourceRef:
+        kind: GitRepository
+        name: holiday-peak-gitops
+        namespace: flux-system
+      interval: 5m
+  upgrade:
+    remediation:
+      retries: 3
+  install:
+    createNamespace: false
+    remediation:
+      retries: 3
+  values:
+    serviceName: crm-profile-aggregation
+    serviceAccount:
+      create: true
+      clientId: 036f4fd2-9093-4948-b0dd-beb5c87aa6dc
+    image:
+      repository: holidaypeakhub405devacr.azurecr.io/holiday-peak-hub/crm-profile-aggregation-dev
+      tag: azd-deploy-1775344626
+    replicaCount: 2
+    resources:
+      limits:
+        cpu: 500m
+        memory: 512Mi
+      requests:
+        cpu: 250m
+        memory: 256Mi
+    nodeSelector:
+      agentpool: agents
+    tolerations:
+    - effect: NoSchedule
+      key: workload
+      operator: Equal
+      value: agents
+    availability:
+      strategy:
+        type: RollingUpdate
+      rollingUpdate:
+        maxUnavailable: 25%
+        maxSurge: 25%
+    pdb:
+      enabled: false
+    keda:
+      enabled: false
+    agc:
+      enabled: true
+      gatewayClassName: azure-alb-external
+      hostnames:
+      - esbcc8bcfyazbbdg.fz03.alb.azure.com
+      parentRefs:
+      - name: holiday-peak-agc
+        namespace: holiday-peak-crud
+      paths:
+      - path: /crm-profile-aggregation
+        pathType: PathPrefix
+        rewriteTo: /
+    env:
+      AI_SEARCH_AUTH_MODE: managed_identity
+      AI_SEARCH_ENDPOINT: https://holidaypeakhub405devsearch.search.windows.net
+      AI_SEARCH_INDEX: catalog-products
+      AI_SEARCH_INDEXER_NAME: search-enriched-products-indexer
+      AI_SEARCH_VECTOR_INDEX: product_search_index
+      APPLICATIONINSIGHTS_CONNECTION_STRING: InstrumentationKey=d2fb45bc-e648-4da2-9345-d278636aede5;IngestionEndpoint=https://centralus-2.in.applicationinsights.azure.com/;LiveEndpoint=https://centralus.livediagnostics.monitor.azure.com/;ApplicationId=d8eb64c4-956d-46ab-a02e-d481deadaa0b
+      AZURE_CLIENT_ID: 036f4fd2-9093-4948-b0dd-beb5c87aa6dc
+      AZURE_TENANT_ID: 16b3c013-d300-468d-ac64-7eda0820b6d3
+      BLOB_ACCOUNT_URL: https://holidaypeakhub405devstor.blob.core.windows.net
+      BLOB_CONTAINER: agent-memory
+      CATALOG_SEARCH_REQUIRE_AI_SEARCH: 'true'
+      COSMOS_ACCOUNT_URI: https://holidaypeakhub405-dev-cosmos.documents.azure.com:443/
+      COSMOS_CONTAINER: agent-memory
+      COSMOS_DATABASE: holiday-peak-db
+      EMBEDDING_DEPLOYMENT_NAME: text-embedding-3-large
+      EVENT_HUB_NAMESPACE: holidaypeakhub405-dev-eventhub
+      FOUNDRY_AGENT_NAME_FAST: crm-profile-aggregation-fast
+      FOUNDRY_AGENT_NAME_RICH: crm-profile-aggregation-rich
+      FOUNDRY_AUTO_ENSURE_ON_STARTUP: 'true'
+      FOUNDRY_STREAM: 'true'
+      AGENT_FOUNDRY_INVOKE_TIMEOUT_SECONDS: '60'
+      FOUNDRY_STRICT_ENFORCEMENT: 'false'
+      KEY_VAULT_URI: https://holidaypeakhub405-dev-kv.vault.azure.net/
+      MODEL_DEPLOYMENT_NAME_FAST: gpt-5-nano
+      MODEL_DEPLOYMENT_NAME_RICH: gpt-5
+      POSTGRES_AUTH_MODE: entra
+      POSTGRES_DATABASE: holiday_peak_crud
+      POSTGRES_HOST: holidaypeakhub405-dev-postgres.postgres.database.azure.com
+      POSTGRES_SSL: 'true'
+      POSTGRES_USER: holidaypeakhub405-dev-crud-identity
+      PROJECT_ENDPOINT: https://holidaypeakhub405devais.services.ai.azure.com/api/projects/aipholidaris
+      PROJECT_NAME: aipholidaris
+      REDIS_HOST: holidaypeakhub405-dev-redis.redis.cache.windows.net
+      REDIS_PASSWORD_SECRET_NAME: redis-primary-key