[Bug]Project registration initialization edgecases

[alexandreLamarre]

Problem

There are edgecases that fail silently during project registration initialization and as well inconsistent behaviour in label handling in the HPO namespace controller.

The problem is control flow based:

• This code must run whether or not we're in a valid project-registration namespace
  • for label cleanup
  • for enforcing label correctness -- i.e. projectId's must match between implementation upstream project labels & HPO project labels
• That code block on considers a hardcoded field.cattle.io/projectId
• It returns nil on error, which means it does not fail loudly in the initialization block
  • This is important because resource conflicts from users/ other controllers updating namespaces will fail silently on initialization -- a race condition!

	err := h.initProjectRegistrationNamespaces()
	if err != nil {
		logrus.Fatal(err)
	}

Source

• The initialization list is updated after this function is called (for good reason!), so we are not guaranteed to have our projectRegistrationTracker track the correct namespaces after the above initialization block under two conditions
  • k8s object resource conflicts (or, less importantly, intermittent apiserver issues)
  • a non-standard field.cattle.io/projectId label is being used by end-users

Testcase

A typical regression test of prometheus-federator should suffice to check the changes

[pankajsqe]

Regression of Prometheus-Federator charts is completed on v2.12.1-alpha1.

Details
rancher : v2.12.1-alpha1
rancher-monitoring: 69.8.2-rancher.15
prometheus-federator:107.1.0+up4.1.0-rc.3