Flaky: retry-jitter tests in token/services/utils fail intermittently on CI

[Effi-S]

Fixes #2195

Several tests in token/services/utils/retry_test.go fail intermittently on CI, in both the utest (unit-tests-race) and utest (unit-tests-regression) jobs. They are unrelated to whatever change is being tested, so they turn up as red on arbitrary PRs.

Observed

Two different tests from the same family have failed on consecutive runs of the same branch:

--- FAIL: TestNewRetryRunnerWithJitter_NegativeJitterFactor (0.32s)
    retry_test.go:570:
        Error: Max difference between 2 and 1.2991946530504297 allowed is 0.5,
               but difference was 0.7008053469495703
        Messages: interval 1 should be ~2x interval 0 (no jitter)

--- FAIL: TestNewRetryRunnerWithJitter_NegativeBackoffMultiplier (0.32s)

Affected tests, all in token/services/utils/retry_test.go:

• TestNewRetryRunnerWithJitter_NegativeJitterFactor
• TestNewRetryRunnerWithJitter_NegativeBackoffMultiplier
• likely TestNewRetryRunnerWithJitter_ExcessiveJitterFactor, which asserts on the same quantity

Why they are fragile

Each test drives a real retry runner with a 10 ms initial delay and a 2.0 multiplier, records wall-clock
timestamps around each attempt, then asserts on the ratio between consecutive measured intervals:

ratio := float64(intervals[i]) / float64(intervals[i-1])
assert.InDelta(t, 2.0, ratio, 0.5, "interval %d should be ~2x interval %d (no jitter)", i, i-1)

The intervals being compared are ~10 ms, ~20 ms and ~40 ms. A ratio of two measured durations is
roughly twice as sensitive to scheduler noise as either measurement, and a few milliseconds of delay
landing on one attempt but not its neighbour is enough to leave the ±0.5 band. On a shared runner —
especially under -race, which inflates every timing — that is well within normal variation. The
tolerance is not a safety margin over a known worst case; it is a guess.

Note the tests are also insensitive to uniform slowdown, because stretching every interval equally
leaves the ratio intact. Only a single perturbed interval breaks them, which is exactly why the failures
look random and are hard to reproduce.

Impact

• Red CI on unrelated PRs, which costs a reviewer the time to confirm the failure is spurious, and
  erodes trust in the signal.
• The failure rotates between tests in the family, so it does not look like a single known flake.

Reproduction

Not reproduced locally: 24+ runs of the affected tests, including with -race and under saturating CPU
load (96 busy loops on 32 cores), all passed. The load run took 269 s versus 6 s idle, so the tests were
heavily slowed without failing — consistent with the ratio being robust to uniform slowdown and
sensitive only to a single perturbed interval. Observed so far only on GitHub-hosted runners.

Suggested direction

These tests are trying to verify argument clamping — that a negative backoffMultiplier defaults to
2.0, and a negative jitterFactor clamps to 0.0. That is a pure input-validation property and does not
need a clock at all. Asserting on the runner’s computed delay sequence, rather than on measured
wall-clock time, would test the same behaviour deterministically. If timing coverage is wanted
separately, it should assert a generous lower bound on total elapsed time rather than a ratio between
adjacent samples.