How to make a 'resource' template to get its manifest from a hard-wired Git input artifact?

[Stolz]

My goal is to submit a workflow whose YAML manifest is in a remote Git repository.

Of course I could reinvent the wheel and use multiple steps with custom 'container' templates (one to clone the repo, another to submit the fetched workflow manifest), but to keep things lean instead I would like to leverage Argo Workflows built-in features as much as possible. Namely:

• the hardwired Git artifacts feature and
• the Manifest From Artifact feature of the resource template.

However I cannot find any example of how to use both features combined. Reading the CRDs specs this is the best I've come up with to submit https://github.com/argoproj/argo-workflows/blob/main/examples/hello-world.yaml in a single step:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: submit-workflow-from-git-
  namespace: argo-workflows
spec:
  entrypoint: main
  arguments:
    parameters:
      - name: git-repository
        value: https://github.com/argoproj/argo-workflows.git
      - name: git-branch
        value: main
      - name: path-to-manifest-file
        value: examples/hello-world.yaml
  templates:
    - name: main
      inputs:
        artifacts:
          - name: cloned-repository
            path: /src
            git:
              repo: "{{workflow.parameters.git-repository}}"
              branch: "{{workflow.parameters.git-branch}}"
              singleBranch: true
              depth: 1
      resource:
        action: create
        manifestFrom:
          artifact:
            name: cloned-repository
            path: /src
            subPath: "{{workflow.parameters.path-to-manifest-file}}"

Unfortunately, the above config fails with read /src: is a directory error message.

So my question is, how can I make a 'resource' template to get its manifest from a hard-wired Git input artifact?

[static-moonlight]

In case you use a kustomization.yaml, you could do something like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
[...]
resources:
  - git@gitlab.company.com:path/to/project.git/path/to/resource?ref=master

I use this mechanic in several workflow projects. I'm not 100% sure though, that this is what you wanted.

[fguendling]

I have exactly the same problem as of today.
@static-moonlight I guess this is not a solution, because this would not dynamically create the resource during Workflow runtime, would it?
@Stolz did you manage to find a solution? Or did you "reinvent the wheel", in the end?

[Stolz]

@fguendling I ended up reinventing the wheel. I tried multiple ways to pass parameters from the invoking workflow to the invoked workflow, but nothing seemed to work, so I ended up using a 3 steps workaround.

• First step clones from Git the repo that contains the workflow template I want to invoke and stores it in a volume.
• Second step mounts the volume, patches the YAML file of the workflow template I want to invoke and saves the patched file as an artifact. The patching consists of overwriting the spec.arguments.parameters: field with a new YAML object generated on run time from the invoking workflow spec.arguments.parameters fields.
• Third and last step uses manifestFrom field to reference the artifact from the second step.

Here is a redacted for brevity version of a plausible config (I use Helm to generate my workflow templates):

---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: submit-workflow-from-git
spec:
  entrypoint: main
  arguments:
    parameters: 
      {{- $workflowParameters := list "repository"  "branch"  "file" -}}
      {{- range $workflowParameters }}
      - name: {{ . }}
      {{- end }}
  templates:
    - name: main
      steps:
        - - name: get-workflow-from-git
            templateRef:
              name: git-tools
              template: shallow-clone-repository
            arguments:
              parameters:
                - {name: repository,    value: "{{`{{workflow.parameters.repository}}`}}"}
                - {name: branch,        value: "{{`{{workflow.parameters.branch}}`}}"}
        - - name: patch-workflow
            inline:
              outputs:
                artifacts:
                  - name: workflow-manifest
                    path: ".argo-workflows/{{`{{workflow.parameters.file}}`}}.yaml"
                    archive:
                      none: {}
              script:
                image: busybox
                workingDir: /workspace
                env:
                  - name: FILE
                    value: ".argo-workflows/{{`{{workflow.parameters.file}}`}}.yaml"
                command: [ash, -euo, pipefail]
                source: |
                  echo ===== Generating Workflow global parameters ...
                  cat << 'EOF' > $FILE.tmp
                        {{- range $workflowParameters }}
                        - name: {{ . }}
                          value: "{{`{{workflow.parameters.`}}{{ . }}{{`}}`}}"
                        {{- end }}
                  EOF
                  echo ===== Injecting generated global parameters into Workflow manifest ...
                  sed -i "/__PARAMETERS_PLACEHOLDER__/ r $FILE.tmp" $FILE
                  cat $FILE
                volumeMounts:
                  - name: workspace
                    mountPath: /workspace
        - - name: submit-workflow
            arguments:
              artifacts:
                - name: workflow-manifest
                  from: "{{`{{steps.patch-workflow.outputs.artifacts.workflow-manifest}}`}}"
              #parameters: (Disabled because it does not seem to work with `manifestFrom`)
            inline:
              inputs:
                artifacts:
                  - name: workflow-manifest
                    path: /tmp/manifest.yaml # Not customizable since `resource.manifestFrom.manifestFrom` has this path hardcoded
              resource:
                action: create
                manifestFrom:
                  artifact:
                    name: workflow-manifest

[fguendling]

Thanks @Stolz we'll also go for that approach.