Description

As it currently stands, all providers would allow all pipeline compiler frameworks, even if the requested provider does not support the requested framework. We should allow the Provider to define what frameworks it supports.

We have also realised that the controller manager should not store any framework-specific config. We currently have the images for the given framework set up in KfpControllerConfigSpec. These should be moved to the Provider CRD, so that the controller manager does not store anything framework specific.

Also, defaultBeamArgs is set at the top of the provider CRD, meaning it is global for all frameworks, despite only being used by the TFX compiler framework. This needs tidying up.

Tasks

• Change the provider CRD to include:

frameworks:
- name: kfpsdk
  image: url to image
  patches:
  - type: json|merge
    patch: “some json patch string”
- name: tfx
  image: url to image
  patches:
  - type: json|merge
    patch: “some json patch string”

• The patch types align with kubectl patch, the patch itself will always be a JSON payload

• The JSON patches need to be applied by the pipeline reconciler to the framework parameters of the pipeline resource JSON before passing to the workflow

• Example of patch with before and after of pipeline frameworks JSON object:

Patch JSON:

[
    {
      "op": "add",
      "path": "/Framework/parameters/beamArgs/-", # Adds to the end of the list
      "value": {
        "name": "newArg",
        "Value": "newValue"
      }
    }
]

Before:

{
  "framework": {
    "type": "tfx",
    "parameters": {
      "components": "base_pipeline.create_components"
    }
  }
}

After:

{
  "framework": {
    "type": "tfx",
    "parameters": {
      "components": "base_pipeline.create_components",
      "newArg": "newValue"
    }
  }
}

and drop DefaultBeamArgs

• Add conversion webhook for provider crd
• Convert from v1alpha5/6 -> v1beta1 and back. Move default beam args to tfx supported framework.
• Add validation to pipeline reconciler to ensure provider supports pipeline resources framework
• In pipeline reconciler at call to trigger workflow need to apply the patches to the pipeline frameworks json
• change compiler code to use new parameters
• update documentation