feat(k8s): graceful-lifecycle hardening -- startupProbe, preStop, drain-readiness (P0-2/3/4/5)

[ELares]

The remaining k8s-readiness P0 fixes (after #743's OOM guard). Plan: deploy/K8S_READINESS_PLAN.md.

Changes

• P0-5 (server, metrics_http.rs + main.rs): /readyz returns 503 the moment graceful shutdown begins, so k8s deprograms this pod's endpoint before the drain -- no client routed to a draining node. MetricsState gains an optional shutdown flag (set.shutdown_flag(), the bug: graceful shutdown can hang joining a shard with a parked subscriber connection (flaky keyspace_notifications teardown) #543 Arc<AtomicBool>); readiness() checks it first. Unit-tested (ready -> drain flips /readyz 200 -> 503 draining).
• P0-2 (chart): startupProbe on /readyz suspends liveness+readiness until load-on-boot completes -- a slow multi-GB snapshot RELOAD no longer trips the ~65s liveness budget and CrashLoopBackOffs a healthy node. failureThreshold*periodSeconds (default 60*10 = 10 min) is a values knob. C3 verified: the metrics listener binds while load-on-boot runs in the shard tasks, so /readyz reports load-incomplete during the reload.
• P0-3 (chart): preStop lame-duck via the native SleepAction (k8s 1.29+) -- NOT exec sleep, which would fail on the shell-free distroless image. Gated on preStop.sleepSeconds > 0 (0 disables it on older k8s -> rely on P0-5). kubeconform -strict @ KUBE_VERSION 1.30 accepts it (SleepAction GA).
• P0-4 (chart): terminationGracePeriodSeconds documented to cover preStop + drain + save-on-exit, sized from the same dataset estimate as the startupProbe budget.

Applied to BOTH the Helm chart (values-tunable) and deploy/k8s/ironcache.yaml (static, parity).

Tests

/readyz-drain unit test + all metrics (17) & shutdown (9) tests green, clippy -D warnings, fmt, dash clean. deploy-lint (helm template | kubeconform -strict @ 1.30) exercises the rendered chart.

🤖 Generated with Claude Code

[ELares]

2-lens review: both SHIP; findings folded in

Lens 1 (server correctness): SHIP, no defects. Decisively confirmed the load-bearing question -- /readyz reads the SAME Arc<AtomicBool> that SIGTERM (apply_signal_flag->flag.store(true)), shutdown_and_join, and committed-cutover all set (single creation at bootstrap.rs:295), so P0-5 is live, not inert. Also verified: flag set at the very start of shutdown (ahead of the drain); /livez untouched so no SIGKILL risk; C3 async-load claim holds (metrics binds while load runs in the shard tasks); the drain test is a real assertion.

Lens 2 (chart/k8s): SHIP. kubeconform-pass verified (1.30 schema has SleepAction). Findings folded in:

• [MEDIUM] preStop defaulted ON would be API-server-rejected on k8s < 1.29. Now gated on semverCompare ">=1.29-0" .Capabilities.KubeVersion so the chart self-protects (degrades to the /readyz-on-drain fallback) instead of a broken install. deploy-lint's helm template now pins --kube-version "${KUBE_VERSION}" so the gated block deterministically renders + is kubeconform-validated.
• [LOW] values.yaml notes the startupProbe budget must cover leader-election time in raft mode too.
• Corrected the SleepAction version phrasing (alpha 1.29 / beta+default-on 1.30 / GA 1.34).

All metrics (17) + shutdown (9) tests, clippy -D warnings, fmt, dash clean.

[github-actions]

perf-gate (A5)

Same-runner ratchet of HEAD against the merge-base (both rebuilt and measured in this job).
PASS = within the noise band, WARN = a real move inside budget (does not fail), FAIL = past budget in the bad direction.

metric	base	head	delta%	band	budget	verdict
qps_median (peak)	82612.48	82183.20	-0.52%	+/-5.00%	drop <= 15%	PASS
bytes_per_key int	46.88	47.50	1.32%	det	rise <= 5%	WARN
bytes_per_key embstr	61.38	60.77	-0.99%	det	rise <= 5%	PASS
bytes_per_key raw	332.73	333.43	0.21%	det	rise <= 5%	PASS

Overall: WARN

• qps: noisy on shared CI, so the band comes from the base reps spread (floored at 5%); a drop is only a regression past the 15% budget.
• bytes_per_key: deterministic (allocator-true memmodel), so a tight 5% rise budget; any rise beyond it FAILs.
• Open-loop tails / criterion micro-benches are reported-not-failed (tail noise is high) and are not part of this ratchet.
• An intentional perf trade is landed by raising the relevant budget in this PR with a documented reason (CI never auto-commits a baseline).