Final score is not the average of all scores

[hxt365]

Nomad version

1.9.7

Operating system and Environment details

Issue

The final score displayed by nomad alloc status -verbose <alloc_id> is not the average of all scores. It's the average of non-zero scores.

Reproduction steps

Expected Result

Actual Result

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[tgross]

Hi @hxt365! This is intentional; zero scores are for scoring elements that don't contribute to the score. Is this somehow resulting in behavior you don't expect? If so, it would help to learn what that behavior is.

(Also, if you don't mind, posting screenshots of terminal text makes them very hard to read.)

[hxt365]

Thanks for your prompt reply! @tgross

This is unexpected when I set affinity for my job. In my case, the binpack score of nonpreferred nodes is very high, hence adding the affinity score for the preferred node does not add much value to the final score, hence the preferred node doesn't get selected.
To illustrate, I have 5 nodes with binpack score of 0.996, 0.995, 0.994, 0.99, and 0.962 (this is the preferred node). Now I set an affinity for the preferred node with weight of 100, so the final score of the 5 nodes become 0.996, 0.995, 0.994, 0.99, 0.981, respectively. You can see that the preferred node does not get selected in this case.

[tgross]

Just for clarity, the weight only changes the affinity scoring relative to other affinity blocks. And if you only want the workload to run on the preferred node, you may want to look at constraint blocks rather than affinity.

It might help to diagnose what's going on here to know more about the rest of your setup? Are you using the default scheduler configuration? Do you have other blocks on this job that are influencing scheduling, like spread? Are these nodes unusually large relative to the size of the workload?

[lukepalmer]

I am tracking an issue that I think is related. Consider the following pathological case:

I have two client nodes, A and B. Pretend they both have binpack scores of 0.8.
I have a job spec that has two affinity blocks:

• One with a weight of 100 that does not match A nor B
• A second with a weight of 1 that matches B but not A

When affinity is calculated for A, the affinity value is 0 so affinity is not appended to option.Scores:

nomad/scheduler/rank.go

Line 975 in bee2400

if totalAffinityScore != 0.0 {

When the final score is calculated for A, the divisor is obtained from len(option.Scores)

nomad/scheduler/rank.go

Line 1017 in bee2400

numScorers := len(option.Scores)

When affinity is calculated for B, the affinity value is 1 out of a possible 101, so the affinity score is about 0.01. Note that the job expresses a small but positive affinity for B. When the final score is calculated for B, the divisor obtained from len(option.Scores) is 1 larger than it is for A. This results in wrong operation.

Oversimplifying the calculation, the final scores end up being calculated like this:
A : 0.8 / 1 = 0.8
B : (0.8 + 0.01) / 2 = 0.4

So, even though the job expressed a positive affinity for node B, this node actually got a lower score. This is wrong.

I haven't tested this yet, but I think the solution is that the exclusion of zeros on line 975 should just be removed.

[tgross]

Hi @lukepalmer! Thanks for that example pathological case, that's very helpful. It does look like that exclusion of zeros is the cause of that, but that's a very old area of the code so I want to do some testing to make sure I understand why that was originally excluded.

I've got a draft PR up here to anchor a discussion about this with some colleagues: #25800