Fix Fleet post renderer errors with non-trivial kustomize git repo

[ron1]

Fleet reports post renderer errors with various non-trivial kustomize git repos such as this which includes modifications to the multi-cluster kustomize example to mimic various real-world kustomize repos.

Steps to reproduce the problem:

1. git clone -b expose-fleet-mc-kustomize-example-post-renderer-bug https://github.com/ron1/fleet-examples.git
2. cd fleet-examples/multi-cluster/kustomize
3. kustomize build overlays/dev - verifies that kustomize successfully builds the dev overlay in this repo
4. fleet test -t dev - reports the following unexpected error:

# Matched: dev
FATA[0000] error while running post render on files: obj '{"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"labels": {"app": "catalog-operator"},
    "name": "catalog-operator", "namespace": "olm"}, "spec": {"replicas": 1, "selector": {
      "matchLabels": {"app": "catalog-operator"}}, "strategy": {"type": "RollingUpdate"},
    "template": {"metadata": {"labels": {"app": "catalog-operator"}}, "spec": {"containers": [
          {"args": ["-namespace", "olm", "-configmapServerImage=quay.io/operator-framework/configmap-operator-registry:latest",
              "-util-image", "quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607"],
            "command": ["/bin/catalog"], "env": "image", "quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607": "imagePullPolicy",
            "IfNotPresent": "livenessProbe", ? {"httpGet": {"path": "/healthz", "port": 8080}}
            : "name", "catalog-operator": "ports", ? [{"containerPort": 8080}, {"containerPort": 8081,
                "name": "metrics", "protocol": "TCP"}] : "readinessProbe", ? {"httpGet": {
                "path": "/healthz", "port": 8080}} : "resources", ? {"requests": {
                "cpu": "10m", "memor' at path 'spec/template/spec/containers/env/valueFrom/configMapKeyRef/name': visit traversal on path: [env valueFrom configMapKeyRef name]: expected sequence or mapping node

Note that when I define a GitRepo for this branch with path multi-cluster/kustomize, its status.resourceErrors is updated as listed below.

status:
  resourcesErrors:
  - |-
    error while running post render on files: obj '{"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"labels": {"app": "catalog-operator"},
    "name": "catalog-operator", "namespace": "olm"}, "spec": {"replicas": 1, "selector": {
      "matchLabels": {"app": "catalog-operator"}}, "strategy": {"type": "RollingUpdate"},
    "template": {"metadata": {"labels": {"app": "catalog-operator"}}, "spec": {"containers": [
          {"args": ["-namespace", "olm", "-configmapServerImage=quay.io/operator-framework/configmap-operator-registry:latest",
              "-util-image", "quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607"],
            "command": ["/bin/catalog"], "env": "image", "quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607": "imagePullPolicy",
            "IfNotPresent": "livenessProbe", ? {"httpGet": {"path": "/healthz", "port": 8080}}
            : "name", "catalog-operator": "ports", ? [{"containerPort": 8080}, {"containerPort": 8081,
                "name": "metrics", "protocol": "TCP"}] : "readinessProbe", ? {"httpGet": {
                "path": "/healthz", "port": 8080}} : "resources", ? {"requests": {
                "cpu": "10m", "memor' at path 'spec/template/spec/containers/env/valueFrom/configMapKeyRef/name': visit traversal on path: [env valueFrom configMapKeyRef name]: expected sequence or mapping node
  - |-
    error while running post render on files: obj '{"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"labels": {"app": "catalog-operator"},
    "name": "catalog-operator", "namespace": "olm"}, "spec": {"replicas": 1, "selector": {
      "matchLabels": {"app": "catalog-operator"}}, "strategy": {"type": "RollingUpdate"},
    "template": {"metadata": {"labels": {"app": "catalog-operator"}}, "spec": {"containers": [
          {"args": ["-namespace", "olm", "-configmapServerImage=quay.io/operator-framework/configmap-operator-registry:latest",
              "-util-image", "quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607"],
            "command": ["/bin/catalog"], "env": "image", "quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607": "imagePullPolicy",
            "IfNotPresent": "livenessProbe", ? {"httpGet": {"path": "/healthz", "port": 8080}}
            : "name", "catalog-operator": "ports", ? [{"containerPort": 8080}, {"containerPort": 8081,
                "name": "metrics", "protocol": "TCP"}] : "readinessProbe", ? {"httpGet": {
                "path": "/healthz", "port": 8080}} : "resources", ? {"requests": {
                "cpu": "10m", "memor' at path 'spec/template/spec/containers/env/valueFrom/configMapKeyRef/name': visit traversal on path: [env valueFrom configMapKeyRef name]: expected sequence or mapping node
  - |-
    error while running post render on files: obj '{"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"labels": {"app": "catalog-operator"},
    "name": "catalog-operator", "namespace": "olm"}, "spec": {"replicas": 1, "selector": {
      "matchLabels": {"app": "catalog-operator"}}, "strategy": {"type": "RollingUpdate"},
    "template": {"metadata": {"labels": {"app": "catalog-operator"}}, "spec": {"containers": [
          {"args": ["-namespace", "olm", "-configmapServerImage=quay.io/operator-framework/configmap-operator-registry:latest",
              "-util-image", "quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607"],
            "command": ["/bin/catalog"], "env": "image", "quay.io/operator-framework/olm@sha256:de396b540b82219812061d0d753440d5655250c621c753ed1dc67d6154741607": "imagePullPolicy",
            "IfNotPresent": "livenessProbe", ? {"httpGet": {"path": "/healthz", "port": 8080}}
            : "name", "catalog-operator": "ports", ? [{"containerPort": 8080}, {"containerPort": 8081,
                "name": "metrics", "protocol": "TCP"}] : "readinessProbe", ? {"httpGet": {
                "path": "/healthz", "port": 8080}} : "resources", ? {"requests": {
                "cpu": "10m", "memor' at path 'spec/template/spec/containers/env/valueFrom/configMapKeyRef/name': visit traversal on path: [env valueFrom configMapKeyRef name]: expected sequence or mapping node

[ron1]

@StrongMonkey I suspect this is a problem with the version of kustomize currently bundled within Fleet v0.3.1. I believe an upgrade to kustomize v3.8.7 with kustomize/api v0.6.5 and kustomize/kyaml v0.9.4 will fix this problem. Any chance the kustomize version bundled within Fleet can be updated in the next Fleet version?

[StrongMonkey]

@ron1 Did you fix the issue eventually? I believe you have to make two things right. First is to make sure you choose fleet-local workspace since the example is in fleet-local workspace. Second is to make sure your cluster has one of label env: dev, env: test or env: prod.

[ron1]

@StrongMonkey This problem occurs even after your suggestions are applied. Have you tried to reproduce the problem using my patched branch? The problem seems related to kustomize reporting WARNINGS for the supplied manifests. The version of kustomize embedded in fleet 0.3.1, namely 3.8.3, reports these warnings. Kustomize 3.8.7 no longer reports warnings. I suspect the warnings being sent to stdout/stderr are confusing the post renderer and thereby corrupting its output. Specifically, the kustomize-generated WARNING appears within the GitRepo status resource error text.

Pls let me know if you are unable to reproduce the problem using my custom branch.

Note that others on slack have apparently encountered this issue as well using different manifests. The kustomize warning is a common theme.

BTW, I opened issue #171 to address the second point in your comment.

[StrongMonkey]

@ron1 This should be fixed in the latest v0.3.4 by #262. Free free to open it if you are seeing it again.