v0.7.0

[aidandunlop]

Provider CRD & support for multiple compiler frameworks

Summary

This release introduces the Provider Custom Resource, allowing providers to be individually configured and their use restricted. Provider resources own and manage a Provider Service, which makes calls to individual providers and receives run completion events, removing a large portion of the argo workflow templates in the process.
Another key feature of this release is the introduction of support for multiple pipeline frameworks, notably KFP SDK, so users are no longer restricted to only using TFX.

---

What's Changed

Notable updates:

• Define new Provider CRD and use within operator by @Ankitha-Krisshnan, @grahamia, @alexgeorgousis, @TobyPinfold in Define new Provider CRD and use within operator #353
• KFP Operator run completion feed service by @grahamia, @aidandunlop in Operator run completion feed service #368
• Implement a gRPC event trigger to publish events to NATS by @TobyPinfold, @Ankitha-Krisshnan, @chr12c in Implement a gRPC event trigger to publish events to NATS #377
• Replace Argo Events stack with Provider Service component by @alexgeorgousis, @TobyPinfold, @peter-hazell, @grahamia, @chr12c, @aidandunlop in Replace Argo Events stack with Provider Service component #391
• Implement controller to manage Provider service Deployment by @alexgeorgousis, @TobyPinfold, @aidandunlop, @peter-hazell, @grahamia in Implement controller to manage Provider service Deployment #408
• Add multi-compiler functionality by @grahamia, @chr12c, @aidandunlop in Add multi-compiler functionality #483
• Implement KFP SDK compiler by @PeterFras, @paoloambrosio-skyuk, @jmendesky, @edinekj, @Ankitha-Krisshnan, @aidandunlop in Implement KFP SDK compiler #521
• Remove expired start time from VAI schedules by @TobyPinfold, @aidandunlop in Remove expired start time from VAI schedules #583
• Support cross-namespace Pipeline Dependencies by @aidandunlop, @TobyPinfold in Support cross-namespace Pipeline Dependencies #676
• Run timestamps in run completion events by @TobyPinfold, @grahamia in Training time in run completion events #689
• Several @dependabot upgrades

Full Changelog: v0.6.0...v0.7.0

---

Migration

1. This release will require the deployment of a Provider resource configured for the provider of your choosing.
2. If you want to deploy the provider to a separate namespace for security purposes, ensure all RBAC permissions are setup accordingly, following the steps under Role-based access control (RBAC) for providers in https://sky-uk.github.io/kfp-operator/docs/getting-started/installation
3. Be sure to update the operator's configuration (either via Helm or Kustomize) to include the manager.runcompletionWebhook,runcompletionEventTrigger and provider configuration blocks.
4. Each user resource (Pipelines, RunConfigurations, Experiment) will then need to be updated to point to the Provider resource. See the user migration guides for more detail on how to do this.

Caution

When upgrading, all resources will reconcile causing any RunConfigurations using the onChange trigger to start new runs in the given provider

---

This discussion was created from the release v0.7.0.