Several tests in token/services/utils/retry_test.go fail intermittently on CI, in both the utest (unit-tests-race) and utest (unit-tests-regression) jobs. They are unrelated to whatever change is being tested, so they turn up as red on arbitrary PRs.
Observed
Two different tests from the same family have failed on consecutive runs of the same branch:
--- FAIL: TestNewRetryRunnerWithJitter_NegativeJitterFactor (0.32s)
retry_test.go:570:
Error: Max difference between 2 and 1.2991946530504297 allowed is 0.5,
but difference was 0.7008053469495703
Messages: interval 1 should be ~2x interval 0 (no jitter)
--- FAIL: TestNewRetryRunnerWithJitter_NegativeBackoffMultiplier (0.32s)
Affected tests, all in token/services/utils/retry_test.go:
TestNewRetryRunnerWithJitter_NegativeJitterFactorTestNewRetryRunnerWithJitter_NegativeBackoffMultiplier- likely
TestNewRetryRunnerWithJitter_ExcessiveJitterFactor, which asserts on the same quantity
Why they are fragile
Each test drives a real retry runner with a 10 ms initial delay and a 2.0 multiplier, records wall-clock
timestamps around each attempt, then asserts on the ratio between consecutive measured intervals:
ratio := float64(intervals[i]) / float64(intervals[i-1])
assert.InDelta(t, 2.0, ratio, 0.5, "interval %d should be ~2x interval %d (no jitter)", i, i-1)
The intervals being compared are ~10 ms, ~20 ms and ~40 ms. A ratio of two measured durations is
roughly twice as sensitive to scheduler noise as either measurement, and a few milliseconds of delay
landing on one attempt but not its neighbour is enough to leave the ±0.5 band. On a shared runner —
especially under -race, which inflates every timing — that is well within normal variation. The
tolerance is not a safety margin over a known worst case; it is a guess.
Note the tests are also insensitive to uniform slowdown, because stretching every interval equally
leaves the ratio intact. Only a single perturbed interval breaks them, which is exactly why the failures
look random and are hard to reproduce.
Impact
- Red CI on unrelated PRs, which costs a reviewer the time to confirm the failure is spurious, and
erodes trust in the signal.
- The failure rotates between tests in the family, so it does not look like a single known flake.
Reproduction
Not reproduced locally: 24+ runs of the affected tests, including with -race and under saturating CPU
load (96 busy loops on 32 cores), all passed. The load run took 269 s versus 6 s idle, so the tests were
heavily slowed without failing — consistent with the ratio being robust to uniform slowdown and
sensitive only to a single perturbed interval. Observed so far only on GitHub-hosted runners.
Suggested direction
These tests are trying to verify argument clamping — that a negative backoffMultiplier defaults to
2.0, and a negative jitterFactor clamps to 0.0. That is a pure input-validation property and does not
need a clock at all. Asserting on the runner’s computed delay sequence, rather than on measured
wall-clock time, would test the same behaviour deterministically. If timing coverage is wanted
separately, it should assert a generous lower bound on total elapsed time rather than a ratio between
adjacent samples.
Several tests in
token/services/utils/retry_test.gofail intermittently on CI, in both theutest (unit-tests-race)andutest (unit-tests-regression)jobs. They are unrelated to whatever change is being tested, so they turn up as red on arbitrary PRs.Observed
Two different tests from the same family have failed on consecutive runs of the same branch:
Affected tests, all in
token/services/utils/retry_test.go:TestNewRetryRunnerWithJitter_NegativeJitterFactorTestNewRetryRunnerWithJitter_NegativeBackoffMultiplierTestNewRetryRunnerWithJitter_ExcessiveJitterFactor, which asserts on the same quantityWhy they are fragile
Each test drives a real retry runner with a 10 ms initial delay and a 2.0 multiplier, records wall-clock
timestamps around each attempt, then asserts on the ratio between consecutive measured intervals:
The intervals being compared are ~10 ms, ~20 ms and ~40 ms. A ratio of two measured durations is
roughly twice as sensitive to scheduler noise as either measurement, and a few milliseconds of delay
landing on one attempt but not its neighbour is enough to leave the ±0.5 band. On a shared runner —
especially under
-race, which inflates every timing — that is well within normal variation. Thetolerance is not a safety margin over a known worst case; it is a guess.
Note the tests are also insensitive to uniform slowdown, because stretching every interval equally
leaves the ratio intact. Only a single perturbed interval breaks them, which is exactly why the failures
look random and are hard to reproduce.
Impact
erodes trust in the signal.
Reproduction
Not reproduced locally: 24+ runs of the affected tests, including with
-raceand under saturating CPUload (96 busy loops on 32 cores), all passed. The load run took 269 s versus 6 s idle, so the tests were
heavily slowed without failing — consistent with the ratio being robust to uniform slowdown and
sensitive only to a single perturbed interval. Observed so far only on GitHub-hosted runners.
Suggested direction
These tests are trying to verify argument clamping — that a negative
backoffMultiplierdefaults to2.0, and a negative
jitterFactorclamps to 0.0. That is a pure input-validation property and does notneed a clock at all. Asserting on the runner’s computed delay sequence, rather than on measured
wall-clock time, would test the same behaviour deterministically. If timing coverage is wanted
separately, it should assert a generous lower bound on total elapsed time rather than a ratio between
adjacent samples.