Combining sidecars

[verult]

Opening this issue to discuss the feasibility, benefits, and drawbacks of combining CSI controller sidecars and maintaining less repos and images.

One downside of the current "microservices" model of maintaining one controller per repo/image is that an update to a common library, csi-lib-utils for example, requires manual changes that are sometimes duplicated across multiple controllers. One possible solution is to combine controllers into one binary, modeling after kube-controller-manager.

Combining sidecars is non-trivial and would require a strong signal that the improvements are worth the engineering effort. To make such a signal more visible, this bug could be used to collect potential pain points in the current model that are hard to address by other means.

AFAIK there has been some thoughts and work in this area in the past. If you have thought about this, let us know what you think. Thanks!

/cc @chrishenzie

[msau42]

Here's reasons I can think of:

Pros of combining sidecars:

• can use shared informers
• can consolidate common parameters like --leader-election where we've had consistency issues in the past
• eliminate golang binary size overhead
• updating common libraries simpler
• combine metrics, health ports
• reduced release toil
• less confusion for driver authors on which combination of sidecars to use

Cons/challenges:

• How to organize sidecar-specific flags? Do we want to provide fine-grained control of worker threads, timeouts, etc per controller?
• How to hande repos like snapshotter which is more than a sidecar
• Need ability to turn on/off controllers based on what the driver supports, testing matrix, rbac managment
• Development of experimental/alpha features constrained by mature sidecars (probably solvable with feature gating)
• We still need separate common library repos and tooling since csi drivers are using them

[xing-yang]

Instead of going from one sidecar per repo to all sidecars in one repo, should we also consider something in the middle where we combine some sidecars into one repo but leave others in their own repos?

[lpabon]

This something we discussed way back in Kubecon San Diego 2019 F2F. Here is an (now old) example: https://github.com/lpabon/csi-controller

[pohly]

We still need separate common library repos and tooling since csi drivers are using them

Those could be included in a single repo via individual go.mod files in sub-directories.

Let me add another Pro:

• Storage capacity tracking can react to all operations that may affect cacpacity, like resizing. Currently it is limited to volume provisioning and deletion.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[chrishenzie]

/remove-lifecycle stale

[mauriciopoppe]

Instead of going from one sidecar per repo to all sidecars in one repo, should we also consider something in the middle where we combine some sidecars into one repo

Another approach I've seen is a single repo with multiple packages like what https://lerna.js.org/ manages, we'd keep the source code of each project isolated in its own directory (minus the common code which would be another package in the repo) and still having the same artifacts as if they were on their own.

In lerna versioning has two modes, fixed where there's a single shared version for all the packages (like a repo version) and independent where each package can have its own release cycle. We could have something similar for go modules.

The directory structure could be something like:

.cloudbuild.yml
Makefile
release-tools/
packages/
  csi-lib-utils/
  external-attacher/
  external-provisioner/
  external-resizer/
  external-snapshotter/

The sidecars would refer to csi-lib-utils with a go replace line in their go.mod file.

Pros:

• In the fixed mode we could test all of these sidecars as a single unit, that would echo on @msau42's less confusion for driver authors on which combination of sidecars to use because there would be a single version for all of them

Cons:

• Building, I don't think it'd be feasible to build the projects in serial, probably should consider building with make -j
• Versioning, in the fixed mode if there's a fix in one package that means a bump in all of the other ones

Storage capacity tracking can react to all operations that may affect capacity, like resizing. Currently it is limited to volume provisioning and deletion.

With a single repo we could test both images (provisioner and resizer) on presubmit

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten