[do-not-merge][experiment]: asynchronous watch registration

[a-hilaly]

Previously, when a ResourceGraphDefinition had issues (e.g., missing RBAC
permissions), the Register() method would block indefinitely on
cache.WaitForCacheSync() while holding a global lock, causing all other RGDs
to stall. This change makes registration non-blocking by moving the cache sync
wait into a goroutine and using channel-based notification when sync completes.

/hold

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: a-hilaly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [a-hilaly]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tjamet]

IIRC the problem is then RGDController tries to register an instance controller to the dynamicController, it blocks any other RGD reconciliation because the dynamic controller holds a global lock that never gets released because lazyInforner.AddHandler never returns waiting for cache sync that never happens, correct?

In the current implementation, there are 2 identified cases:

• failing to start any parent informer (because permission error)
• failing to start any child informer (because permission error)

We should recover when:

• KRO got granted permissions to list/watch the parent resource
• KRO got granted permissions to list/watch the child resource
• The RGD got updated and removes the need to watch resources without permissions

From quick read, I we detect permission changes that would recover a functional state.
It looks like we are already watching for informer errors.
Would updating this handler for something like

	_ = inf.SetWatchErrorHandler(func(_ *cache.Reflector, err error) {
		w.log.V(1).Error(err, "watch error for lazy informer", "gvr", w.gvr)
		if isNoPermissions(err) {
			w.startErr = err
			w.Shutdown()
		}
	})

and update the handler registration:

		if !cache.WaitForCacheSync(w.done, w.informer.HasSynced) {
			w.log.Error(fmt.Errorf("cache sync failed"), "lazy informer sync failure", "gvr", w.gvr)
			if w.cancel!= nil{
				w.cancel()
			}
			w.cancel = nil
			w.informer = nil
			w.handlers = make(map[string]cache.ResourceEventHandlerRegistration)
			if w.startErr != nil {
				return w.startErr
			}
			return fmt.Errorf("failed to sync informer for %s", w.gvr)
		}

We would also need to make sure this error is correctly propagated to the RGD reconciler and the RGD status is updated as expected

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.