Shaping the Future Together: Share Your Wishlist for Addon-Controller Features

[gianlucam76]

Hi Sveltos community!

We're constantly striving to improve Sveltos and deliver the best possible experience for managing your Kubernetes add-ons across multiple clusters. Your feedback is invaluable in guiding our development efforts, and we're eager to hear directly from you.

What we're asking:

• What features are most important to you at this stage?
• Are there any specific pain points you'd like to see addressed?
• Do you have any innovative ideas for enhancing Sveltos's capabilities?

Feel free to share any thoughts, suggestions, or feature requests you may have. We encourage detailed descriptions, use cases, and even mockups if you have them!

[oliverbaehler]

I would wish for a strong integration with FluxCD. In my environments ii have everything with fluxCD iand i would like to leverage the capabilities provided by the kustomize-controller and helm-controller.

Btw. in vcluster scenarios, can you deploy within the cluster and in the cluster namespace? That's a common one when staging vclusters.

[gianlucam76]

Thank you @oliverbaehler
Can you give me a concrete example of integration with the Kustomize controller for instance?

So far Sveltos integrates with Flux by using as solution for gitops approach. Meaning Flux sync from git repository, then Sveltos takes it from there and deploy using Kustomize sdk (and so replacing the Flux Kustomize controller). But I would love to hear more uses cases on how sveltos can integrate with Flux.

Regarding the vcluster, I will try it out tomorrow.

Thank you again!

[oliverbaehler]

for me, the dependency management is difficult. Meaning the order of which helm charts are deployed matters. (helmRelease has a depends_on property). I need a property to ignore the changes on sync, because they are manipulated (helmreleases have that property as well).

For the vcluster scenario, how would you build that? It's mainly relevant for the CNI which is going to be running on the cluster. Because otherwise if you eg. deploy helmreleases to the vcluster, they can't be executed. But thinking about it, it's anyway different with sveltos 🤔 nvm

btw can you register clusters in a kubernetes native way?

[gianlucam76]

Thank you @oliverbaehler.

With respect to order, I agree that it matters. And Sveltos honours that in different ways:

When you create a ClusterProfile/Profile and you list multiple helm charts those are deployed in the same exact order. For instance creating this ClusterProfile, in every matching cluster, grafana will be deployed only after prometheus is correctly deployed.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: prometheus-grafana
spec:
  clusterSelector: env=fv
  helmCharts:
  - repositoryURL:    https://prometheus-community.github.io/helm-charts
    repositoryName:   prometheus-community
    chartName:        prometheus-community/prometheus
    chartVersion:     23.4.0
    releaseName:      prometheus
    releaseNamespace: prometheus
    helmChartAction:  Install
  - repositoryURL:    https://grafana.github.io/helm-charts
    repositoryName:   grafana
    chartName:        grafana/grafana
    chartVersion:     6.58.9
    releaseName:      grafana
    releaseNamespace: grafana
    helmChartAction:  Install

ClusterProfiles support dependency management. A ClusterProfile can specify another ClusterProfile (e.g., "kyverno") as a dependency. The deployment of the dependent ClusterProfile will only begin after the successful deployment of all add-ons and applications listed in the dependency.

Here is an example

  apiVersion: config.projectsveltos.io/v1alpha1
  kind: ClusterProfile
  metadata:
    name: kyverno-policies
  spec:
    clusterSelector: env=fv
    dependsOn:
    - kyverno
    policyRefs:
    - deploymentType: Remote
      kind: ConfigMap
      name: disallow-latest-tag
      namespace: default
      kind: ConfigMap
      name: restrict-wildcard-verbs
      namespace: default

With respect to ignore changes on sync, that is a good idea. And I guess you want this per field (so for instance you deploy a deployment but the replicas is managed by Autoscaler and you don't want the replicas to be re-written on sync right)?

[oliverbaehler]

@gianlucam76 Thanks for the response. You have convinced me enough so i will be implementing sveltos for our Solution. I will surely have follow-up proposals. 🤔

[gianlucam76]

Looking forward to your proposals! Those will help making sure Sveltos tackles real problems with a correct approach.

[oliverbaehler]

Just to clarify why I am "Misusing" this issue for my onboarding process. We are automating against a non-CAPI environment and there might be some interesting situations which arise from that, which were not yet considered. Let me know if I should move this to slack, but might be interesting for other users.

First Question/Proposal: How do you solve the case, where you have a new cluster registered, let's say this eg.

apiVersion: v1
items:
- apiVersion: lib.projectsveltos.io/v1alpha1
  kind: SveltosCluster
  metadata:
    creationTimestamp: "2024-06-12T09:13:55Z"
    generation: 1
    labels:
      capsule.clastix.io/tenant: xmodels
      env: production
      is-credential-preset: "true"
      overwrite: this
      project-id: xmodels
    name: pdl6npm7qp
    namespace: cluster-pdl6npm7qp
    spec: {}

Now I have seen here (https://projectsveltos.github.io/sveltos/template/template/ - Variables) that you have access to a lot of CAPI attributes. In our case we don't have access but I still need to configure my charts with certain cluster properties.

So there needs to be a way to add more data to sveltosclusters, which are non CAPI provided. A simple approach, adding a extra context field to sveltosclusters, where I can pass my additional information (essentially an interface of values):

apiVersion: lib.projectsveltos.io/v1alpha1
kind: SveltosCluster
metadata:
  creationTimestamp: "2024-06-12T09:13:55Z"
  generation: 1
  labels:
    capsule.clastix.io/tenant: xmodels
    env: production
    is-credential-preset: "true"
    overwrite: this
    project-id: xmodels
  name: pdl6npm7qp
  namespace: cluster-pdl6npm7qp
  uid: 02f9f1d7-b93a-4944-a449-d77c3bb3672d
spec:
  context:
    address:
      ip: 10.123.11.5
      port: 6443
      url: https://pdl6npm7qp.some.cloud:6443
    versions:
      apiserver: 1.28.9
      controlPlane: 1.28.9
      controllerManager: 1.28.9
      oldestNodeVersion: 1.28.9
      scheduler: 1.28.9

Maybe something as generic as might also be interesting:

---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-resources
spec:
  clusterSelector: env=fv
  templateResourceRefs:
  - resource:
      apiVersion: kubermatic.k8c.io/v1
      kind: Cluster
      name: "{{.Cluster.metadata.name}}"
    identifier: "KubermaticCluster"
  policyRefs:
  - deploymentType: Remote
    kind: ConfigMap
    name: sveltos
    namespace: default

Altough that might be different features. This would aim towards having more flexible template sources on cluster basis.

Or am I just missing a trick? :D

[gianlucam76]

Thanks @oliverbaehler. This space is good for such design discussion, so please keep using it.

One comment regarding the templateResourceRefs section

• namespace can be set or left empty (as in your example). If empty the cluster namespace will be used.
• namespace and name can be defined as template and currently use ClusterNamespace and ClusterName (so not .Cluster.metadata.name though I feel it is better to change that and make it consistent with rest of templating). This is needed in case where for instance I need to fetch a cluster wide resource and there is one resource for managed cluster. (please let me know your opinion on this).

  templateResourceRefs:
  - resource:
      apiVersion: kubermatic.k8c.io/v1
      kind: Cluster
      name: "{{.ClusterName}}"

With respect to allow adding extra information to SveltosCluster I am OK with that. If information is static I think it is useful (though it would be good to extend sveltosctl register cluster command so there is no need of multiple steps when registering a cluster.

If the information is instead dynamic (like anything that depends on Kubernetes version and so can change if cluster is upgraded), if it is possible to get this information in the managed cluster we could use sveltos event framework. So Sveltos will collect from managed cluster and report back to the management cluster. And if that changes, Sveltos automatically updates it (and templating automatic react to such change already). So we have no stale information nor we require user to manually change it (which user might forget to do).

In other words, I am fine adding a generic field in SveltosCluster Spec (maybe a map[string]string) but also present in a way that let user know to add static information there and use sveltos event framework whenever possible for dynamic information.

What do you think?