Move steps extensions to kaoto-ui

[lordrip]

Context

The goal of this discussion is to gauge whether it makes sense or not to move all step extensions from https://github.com/KaotoIO/step-extension-repository into https://github.com/KaotoIO/kaoto-ui

Current status

Whenever we need to create a new step extension, the following steps need to be followed:

1. Create said step extension in the step-extension-repository.
2. Once it's merged, create an entry in the kaoto-viewdefinition-catalog
3. Update the backend with the newly added information

At the moment, this approach it's working but there is the risk of updating a dependency from KaotoUI or from a step extension and turning this link incompatible. Also, we don't have the capability to version said extensions in case of a breaking change.

Short-term proposal

1. Update yarn to v3 in KaotoUI repository
2. Convert KaotoUI into a package of a yarn workspace
3. Bring all step extensions as nested packages of yarn workspaces

Advantages

• A single place to handle dependencies updates
• Whenever a common dependency it's updated, it will be done for all packages at the same time
• Improve the step extensions development by having everything in a single package, so fewer things to bootstrap while developing
• Have a single place for discussions and issues

Long-term proposal

1. Rename the current KaotoUI into KaotoUICore
2. Create a new package called KaotoUI that consumes KaotoUICore and the step extensions as dependencies
3. Turn KaotoUICore into a private package to avoid publishing into NPM
4. Publish the new KaotoUI package to be consumed by KaotoVSCode

Advantages

• Each step extension will be part of the bundle, removing the need to version individual extensions
• Continuing the previous item, we remove the risk of hosting required UI information in a separate server
• Streamline the step extensions development by running only KaotoUICore because whenever another package it's modified, it should be automatically rebuilt.

[lordrip]

cc @kahboom @igarashitm @mmelko @Delawen

[igarashitm]

+1 and this also ease the situation when multiple step extensions want to share some React components, import Kaoto UI core API, etc. We do need this before we add more step extensions.

[kahboom]

Unless I've misunderstood, it seems most of the benefit of the long-term proposal would come with the short-term proposal anyway? Definitely +1 to the short-term proposal, but I'm not sure if the long-term one is necessary?

[lordrip]

Unless I've misunderstood, it seems most of the benefit of the long-term proposal would come with the short-term proposal anyway?

Not exactly,

• Each step extension will be part of the bundle, removing the need to version individual extensions

By having the step extension in the Kaoto UI repo doesn't mean that we're bundling together yet, but that they will share dependencies versions but the link will still happen as module federation. Any KaotoUI version will use the latest version of said step extension.

• Continuing the previous item, we remove the risk of hosting required UI information in a separate server

In the short term, we will still load the extensions from the URL, so technically it will be the same GitHub server, for the long term, we would take these packages and add them to KaotoUI as a dependency, so now every KaotoUI version will have the extensions bundled, without requiring module federation for them. We will keep the module federation feature for other extensions that might come in the future though 😃

[kahboom]

Why would we need to version individual extensions if they are using the same dependencies? How would we then treat third-party step extensions that are stored externally?

because whenever another package it's modified, it should be automatically rebuilt.

But not at runtime, right? We lose benefits without module federation, which, if we decide to do then fine but I think it needs to be clear to the rest of the team what the tradeoffs are, particularly the non-frontend devs.

EDIT: Just to be clear, I'm for the proposal, particularly the short term one, but I would like to see a wider conversation happening around removing module federation for step extensions and what it would mean, since it's a big part of Kaoto.

[igarashitm]

I thought we keep step-extension-repository and module federation for the 3rd party step extensions. Couldn't it be still module federation, but load from sub packages for OOTB step extensions? So that only 3rd party step extensions are loaded from GH pages.
Someday we might even want to override OOTB extensions in the downstream project for customization.

[lordrip]

We lose benefits without module federation

@kahboom the proposal is not about removing module federation, module federation is here to stay 😄 , I'm referring to including the out-of-the-box extensions in the Kaoto bundle. We'll keep module federation to allow 3rd party extension.

I would like to see a wider conversation happening around removing module federation for step extensions and what it would mean, since it's a big part of Kaoto.

That conversation shouldn't happen, as we're not planning to remove module federation 😄, we could discuss whether we want to provide out-of-the-box step extensions in the main bundle though

[lordrip]

First step towards this goal: #2159