`spec.postBuild.substitute` variables not being applied to the content of `spec.patches`

[alanrichman]

We are using Flux 2.7.2 and are seeing a problem where variables declared in spec.postBuild.substitute of a Kustomization are not being applied to the content of spec.patches of that same Kustomization. We use this behavior to extend the usage of ImagePolicy resources since the control comments for them do not work when in a Yaml string literal. We also use it for some other things but this is the easiest example to illustrate.

This worked before the 2.7 release:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: some-tool
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  path: ./path
  postBuild:
    substitute:
      someToolImage: my-registry.com/some-tool # {"$imagepolicy": "flux-system:some-tool:name"}
      someToolTag: 1.2.3 # {"$imagepolicy": "flux-system:some-tool:tag"}
  patches:
    - target:
        kind: HelmRelease
        name: some-tool
      patch: |-
        - op: add
          path: /spec/values/controller
          value:
            image: ${someToolImage}
            tag: ${someToolTag}

Now it produces a manifest in-cluster like this:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: some-tool
  namespace: flux-system
spec:
  patches:
  - patch: "- op: add\n  path: /spec/values/controller/image\n
      \ value:\n    repository: \n    tag: @"
    target:
      kind: HelmRelease
      name: some-tool
  path: ./path
  postBuild:
    substitute:
      someToolImage: my-registry.com/some-tool
      someToolTag: 1.2.3
  sourceRef:
    kind: GitRepository
    name: flux-system

When it reconciles it produces this error which I suspect is just due to the variable not being substituted (or being substituted as an empty string) causing invalid yaml.

unable to parse SM or JSON patch from [patch: "- op: remove\n  path: /spec/chart/spec/version\n- op: replace\n  path: /spec/chart/spec/sourceRef/name\n  value: luna-sts\n- op: add\n  path: /spec/values/imagePullSecrets\n  value:\n    - name: harbor-cluster-docker-secret\n- op: add\n  path: /spec/values/operator/image\n  value:\n    repository: \n    tag: @"]

[stefanprodan]

A Flux Kustomization does not alter itself, you probably have post build enabled in the parent and that has someToolTag: @. You can disable post build on this Flux Kustomization with:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: some-tool
  namespace: flux-system
  annotations:
    kustomize.toolkit.fluxcd.io/substitute: disabled

Also you should enable this from docs:

Note: It is recommended to set the --feature-gates=StrictPostBuildSubstitutions=true controller flag, so that the post-build substitutions will fail if a variable without a default value is declared in files but is missing from the input vars.

[alanrichman]

First I was mistaken when I told you the version. The cluster I was seeing this error on is using v2.6.2 and it was previously working on v2.5.1.

The above example is pseudo-code but here are the functional test manifests I have applied to a brand new cluster:

./clusters/local/infrastructure.yaml
(flux manifests are in ./clusters/local/flux-system)

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: deployment
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  path: ./workloads/deployment
  prune: true
  postBuild:
    substitute:
      nginxImage: nginx:1.29.3
  patches:
    - target:
        kind: Deployment
        name: nginx-deployment
      patch: |-
        - op: replace
          path: /spec/template/spec/containers/0/image
          value: ${nginxImage}

./workloads/deployment/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml

./workloads/deployment/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

The deployment starts using the nginx:1.29.3 image specified in the Kustomization's postBuild by way of its patches. All that to say when I upgraded my test cluster to v2.6.2 then to v2.7.3 it still continued to work as I expected. So I need to check a bit more about why this was failing because I am not currently able to reproduce this bug in my testing environment.

[alanrichman]

Also thanks for the suggestion about the feature gate, I am going to apply that!