Deploy the same repo via kustomize to different namespaces on the same cluster

[stbraley]

My repository is structured as follows. However, when I provide a path to the dev folder. Fleet does not seem to find the base folder? I'd like to use this repo to deploy multiple instances of my app to different namespaces on the same cluster. Currently we are using ArgoCD to do this by pointing it to the overlay folder we want to deploy. For example, the dev folder. When we use Argo, it finds the base folder. In Fleet, however, I can't seem to find a way to do the same thing. I do see a way to use the same repo to deploy to multiple clusters using different overlay folders by using a fleet.yaml file. However, it does not look like the fleet.yaml file will support the same cluster with different namespaces. Am I missing something in the documentation?

Error Produced

error while running post render on files: accumulating resources: accumulateFile "accumulating resources from '../../base': '../../base' doesn't exist", loader.New "error loading ../../base with git: url lacks host: ../../base, dir: '../../base' doesn't exist, get: invalid source string: ../../base"

Repo Structure:

• base
• overlays
  • dev
    • kustomization.yaml
  • stage
  • prod
  • temp

gz#14201

[ron1]

You might try creating an individual fleet.yaml file for each overlay.

[stbraley]

Yes, I've tried. It doesn't work. It looks like Fleet requires the base directory to be a child of the path. If this is true. Then kustomize in fleet won't work for multiple instances in the same cluster. Very disappointing as this is basic 101 type stuff. If I can apply my kustomization.yaml file using the following. Fleet should be able to do the same.

kubectl apply -k overlays\dev -n dev

kustomization.yaml located in overlays\dev

bases:
- ../../base
resources:
  - ingress.yaml
patches:
  - image-tag.yaml
  - replica-count.yaml
configMapGenerator:
- name: myapp
  behavior: replace
  envs:
    - configs/settings.env

[ron1]

It seems like the default Fleet repo scanning algorithm described here: https://github.com/rancher/fleet/blob/master/docs/gitrepo-structure.md#how-repos-are-scanned should be more configurable via the GitRepo resource to allow explicit specification of one or more fleet.yaml file paths. This would allow the Fleet GitRepo resource to behave more like the ArgoCD Application resource to support this type of scenario. This would allow you to create one Fleet GitRepo resource per overlay that explicitly references a single fleet.yaml.

[ron1]

@stbraley Please provide a copy of your GitRepo manifest.

[stbraley]

I just used the UI to build the manifest. But here is what it generated.

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  creationTimestamp: "2021-01-05T20:47:40Z"
  generation: 3
  name: caas-demo
  namespace: apps7
  resourceVersion: "301964103"
  selfLink: /apis/fleet.cattle.io/v1alpha1/namespaces/apps7/gitrepos/caas-demo
  uid: 2b23e586-992a-40a8-ac01-5711c60d15d7
spec:
  branch: fleet
  clientSecretName: my-secret
  forceSyncGeneration: 2
  paths:
  - /caas/overlays/demo
  repo: repourl
  targetNamespace: demo
  targets:
  - clusterSelector:
      matchLabels:
        management.cattle.io/cluster-name: c-vzcs4
status:
  commit: 6b44fec07e1e40dfd53947bda53a48b5e88f7613
  conditions:
  - lastUpdateTime: "2021-01-05T20:50:18Z"
    message: 'ErrApplied(1) [Bundle caas-demo-caas-overlays-demo: error while running
      post render on files: accumulating resources: accumulateFile "accumulating resources
      from ''../../base'': ''../../base'' doesn''t exist", loader.New "error loading
      ../../base with git: url lacks host: ../../base, dir: ''../../base'' doesn''t
      exist, get: invalid source string: ../../base"]'
    status: "False"
    type: Ready
  - lastUpdateTime: "2021-01-05T20:50:18Z"
    status: "True"
    type: Accepted
  - lastUpdateTime: "2021-01-05T20:49:51Z"
    status: "False"
    type: Reconciling
  - lastUpdateTime: "2021-01-05T20:47:40Z"
    status: "False"
    type: Stalled
  - lastUpdateTime: "2021-01-05T20:49:51Z"
    status: "True"
    type: Synced
  desiredReadyClusters: 1
  display:
    error: true
    message: 'error while running post render on files: accumulating resources: accumulateFile
      "accumulating resources from ''../../base'': ''../../base'' doesn''t exist",
      loader.New "error loading ../../base with git: url lacks host: ../../base, dir:
      ''../../base'' doesn''t exist, get: invalid source string: ../../base"'
    readyBundleDeployments: 0/1
    state: ErrApplied
  gitJobStatus: Current
  observedGeneration: 3
  readyClusters: 0
  resourceCounts:
    desiredReady: 0
    missing: 0
    modified: 0
    notReady: 0
    orphaned: 0
    ready: 0
    unknown: 0
    waitApplied: 0
  resourceErrors:
  - 'error while running post render on files: accumulating resources: accumulateFile
    "accumulating resources from ''../../base'': ''../../base'' doesn''t exist", loader.New
    "error loading ../../base with git: url lacks host: ../../base, dir: ''../../base''
    doesn''t exist, get: invalid source string: ../../base"'
  summary:
    desiredReady: 1
    errApplied: 1
    nonReadyResources:
    - bundleState: ErrApplied
      message: 'error while running post render on files: accumulating resources:
        accumulateFile "accumulating resources from ''../../base'': ''../../base''
        doesn''t exist", loader.New "error loading ../../base with git: url lacks
        host: ../../base, dir: ''../../base'' doesn''t exist, get: invalid source
        string: ../../base"'
      name: caas-demo-caas-overlays-demo
    ready: 0
type: fleet.cattle.io.gitrepo

[ron1]

@stbraley Does the fleet example in the following git repo adequately capture your use case: https://github.com/ron1/fleet-examples/tree/fleet-mc-kustomize-multi-namespace-example/multi-cluster/kustomize-multi-namespace?

@StrongMonkey Is the above example a development-centric use case Fleet would aspire to support? Today, we must use ArgoCD deployed via Fleet to support these development-centric, multi-namespace workloads.

[stbraley]

@ron1 I think so with one caveat. It needs to ignore folders in the overlay directories that don't have a path specified in the GitRepo resource. We have some deployments in the git repository that are not intended to be deployed to clusters.

[ron1]

@stbraley @StrongMonkey I made a minor change to the GitRepo in file: https://github.com/ron1/fleet-examples/blob/fleet-mc-kustomize-multi-namespace-example/multi-cluster/kustomize-multi-namespace/README.md to allow multiple fleetFiles per path. A GitRepo path with one or more explicit fleetFiles that include helm and/or kustomize directives should not search for unspecified raw yaml manifests. So, I would expect Fleet to ignore "...deployments in the git repository that are not intended to be deployed to clusters".