Self managed sharded workqueue for scaling controller

[Anuragch]

Current concurrency control in controller framework does not provide a way to dynamically route a specfific subset of events to one of Reconcile goroutine.
Do you see any issue self managing multiple workqueue and use toplevel Reconcile call just to route it to correct queue and let background worker do the actual custom reconciliation logic after pulling items from thier respective queues.

Apart from Retry/Forget what all caveats i should keep in mind?

[camilamacedo86]

Thanks for raising this!

I’m not entirely sure I follow your question fully, so just to clarify how the current model works:

• In the controller-runtime framework, we have a Manager.
• The Manager manages one or more Controllers.
• Each Controller has a Reconciler, which implements the reconciliation loop.
• "This loop is triggered by Kubernetes events (via Watches) on resources that the controller is interested in." ( to make it easier to understand )
• The reconciliation pattern follows the Operator model: it tries to drive the actual state toward the desired state, similar to how Kubernetes works internally. ( see: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ )

I suggest checking these sections of the Kubebuilder book for foundational insights:

• https://book.kubebuilder.io/getting-started
• https://book.kubebuilder.io/cronjob-tutorial/gvks
• https://book.kubebuilder.io/reference/watching-resources

If I misunderstood your intent or setup, feel free to share more details, and I’ll be happy to take another look!

[Anuragch]

Thanks For responding @camilamacedo86
Let me clarify the approach I was suggesting. In the default implementation, informers push events to a workqueue, and based on the concurrency level, multiple goroutines pop events from the queue and call
Reconcile.

My suggestion was to implemented the Reconcile method so that it only reroutes the events to a set of managed queues controlled by the custom controller. The consumers of these queues would then handle the actual reconciliation logic. This approach would allow for dynamic control over how specific sets of events are processed based on selection criteria. Specific set of events can be given more processing bandwidth.

Of course, the new, self-managed reconcile loop would still need to ensure that retries are properly handled. I was wondering what other caveats or potential issues I should consider in this design.

func (c *MyController)Reconcile(ctx, r* Requets) (Result, err) {
      .
      c.someQueue.Add(r)
      return ctrl.Result{}, nil

}

[camilamacedo86]

Hi folks 👋

To avoid triggering reconciliations on all events, you can use predicates in the watch setup:

• Kubebuilder docs: https://kubebuilder.io/reference/watching-resources
• Operator SDK has a good example too: https://sdk.operatorframework.io/docs/building-operators/golang/references/event-filtering/

Hope that helps!

As for changing this behaviour in controller-runtime, I don’t think it's within scope—it’s working as designed. If you feel strongly, you could raise an RFE about it, but your goal might already be achievable with existing code implementation and features (unless I misunderstood).

Also, could you clarify what you'd like from Kubebuilder in this case? We rely on controller-runtime, so defaults likely won’t change—but we’re happy to help if you can share more about what you're expecting.