Split rule.status.NodeEvaluationResults into a separate CRD NodeReadinessEvaluation

[ajaysundark]

We discussed this in the past about eventually splitting the node evaluation results from the rule status object and while I have been going through the sig-arch api-conventions, I found below guidelines-

References in the status of the referee to the referrer may be permitted, when the references are one-to-one and do not need to be frequently updated, particularly in an edge-based manner

Our current usage seem to be in conflict with these guidelines, having 1 to many node references in rule status requiring frequent updates.

While having node results directly in the status is nice for observability, this has caused us some concurrency issues that we fixed in the past.

We need to investigate separating the node references into their own. We could decide this after some user feedback or scalability tests.

This applies to three fields - AppliedNodes, FailedNodes, and NodeEvaluations in NodeReadinessRuleStatus object.

[ajaysundark]

cc @Karthik-K-N

[Karthik-K-N]

Thanks for tagging, I can explore on this share my observation.

[ajaysundark]

/remove-priority backlog

[ajaysundark]

In addition to handling complexity of concurrent updates, storing node-evaluation results in single rule object will likely exceed the size of the etcd 1.5MB limit -- ref: similar reasoning for scaling endpoint slices to pod objects here: https://kubernetes.io/blog/2020/09/02/scaling-kubernetes-networking-with-endpointslices/#scalability-limitations-of-the-endpoints-api

@Karthik-K-N I think we could prioritize this sooner for reliability of the controller in high scale scenarios.

[ajaysundark]

Copying discussions w/ Karthik here for completeness --

I think we still need to think it through.

We'd need to split this into two separate problems:

1. Removing node-references from rule-status: AppliedNodes, FailedNodes and NodeEvaluations.
2. Redesign new CR for NodeEvaluations.

Reading through https://kubernetes.io/blog/2020/09/02/scaling-kubernetes-networking-with-endpointslices/#scalability-limitations-of-the-endpoints-api, I realized  (1) is more of a concern than (2) for scale.

To answer your question on cardinality of NodeEvaluation for (2):

you mean to say 1 result object per node, This will contain all the applicable rules for that node?

we have two options -
[A] 1 node-eval per rule per node, alternatively
[B] 1 node-eval per node (merging all rule results into this single object).

[A] Assuming (1 - 15) rules x 5000 node objects this would raise to (5000 - 75000) eval objects. This will bring in lot of complexity around cleaning up / latency.
[B] 5k eval objects is better, but we are doubling the 'node-like' objects in a cluster. Cleaning up may be manageable with owner-references but I think we may uncover more complexities as we scale test.

We can handle (2) in incremental designs along with planning for scale-tests. We could plan for (1) sooner,

[Karthik-K-N]

I agree, In this case

Removing node-references from rule-status: AppliedNodes, FailedNodes and NodeEvaluations.

How the user is going to know whether the rule is applied to Node or not, Are events sufficient?

[Karthik-K-N]

Given that I have done API changes previously I can take this up, If no one is currently working
/assign