promoter-apiserver should return 410 Gone for unresumable watch resourceVersions

[crenshaw-dev]

Bug

PromotionStrategyDetails is a virtual resource served by promoter-apiserver. Resource versions are synthetic and in-memory: the counter starts at 1 whenever the process starts (internal/apiserver/index.go).

Watch behavior today:

• resourceVersion empty / "0" (or sendInitialEvents) → initial snapshot, then deltas. Correct.
• any other resourceVersion → no snapshot, deltas only. That matches Kubernetes if the server can resume from that RV.
• the server never rejects an RV it cannot resume. After an apiserver restart, a watch at a previous process's RV is accepted (200) and sits empty until a later mutation.

Kubernetes requires the opposite: if history for that RV is gone, the watch must fail with HTTP 410 Gone so the client clears its cache, re-lists, and starts a new watch from the list RV.

From Efficient detection of changes:

When the requested watch operations fail because the historical version of that resource is not available, clients must handle the case by recognizing the status code 410 Gone, clearing their local cache, performing a new get or list operation, and starting the watch from the resourceVersion that was returned.

See also Resource versions. client-go reflectors already treat 410 as expired (IsResourceExpired / IsGone) and re-list; they will not recover if the server returns 200 and never sends the missing history.

The apiserver already closes overflowed watchers so the client re-lists (internal/apiserver/watch_behavior_test.go). Stale/unknown RVs should force the same recovery, via 410.

Expected

Watch (and list with a specific resourceVersion that cannot be served) should return 410 Gone (StatusReasonExpired) when:

• the RV is not from this process (counter reset on start), or
• the RV is otherwise unknown / older than the server can resume.

A watch at the current RV should continue to skip the snapshot and stream only subsequent deltas.

Impact

Any list-then-watch client whose cache outlives an apiserver restart (informers, kubectl --watch, aggregated-API consumers) can keep a stale view until the client process itself restarts or an object mutates.