Make node retirement fit inside the AWS spot notice

[harshit-anyscale]

AWS gives 120 seconds of warning before it reclaims a spot machine. Serve does not get 120 seconds: detection costs up to 10s (a 5s spot-notice poll, then a 5s node-manager loop) and the drain action itself is capped at 60s. Every value below is derived from that budget.

t=0        AWS issues the spot notice          -> machine dies at t=120s
t<=10s     detected, node enters STOPPING
t~10-70s   the 60s drain action - all Serve gets
t=120s     AWS terminates the instance

The timeout ladder

Four timeouts nest, each guarding the one below it:

Layer	Value	What it does
long-runner ceiling	20s	the longest work the service will do
request_timeout_s	26	replica returns 408 past this
HAPROXY_TIMEOUT_SERVER_S	28	HAProxy gives up on the backend, becomes a 500
HAPROXY_HARD_STOP_AFTER_S	30	a soft-stopping worker is SIGKILLed

Read it top down. Hard-stop must be at or under 60s to fit inside the drain action, and it should equal RAY_SERVE_DIRECT_INGRESS_MIN_DRAINING_PERIOD_S (30, the default) so an old HAProxy worker can never outlive the replica it is routing to — closing the previous 30s/400s mismatch. Subtract the two ordering guards and the request ceiling lands at 20s.

request_timeout_s deliberately sits below TIMEOUT_SERVER_S so an overrun surfaces as a clean replica 408 rather than a generic HAProxy 500 — the difference between a diagnosable failure and a mystery.

6s of headroom over the ceiling is generous. Measured transport and queueing overhead above the deliberate sleep was p50 0.3s / p99 0.1s (n=408, 2026-08-28).

What changed

Workload — long_runner.py, traffic_model.py

• long-runner 30–120s → 8–20s, clamp 125 → 20. This is what lets hard-stop reach 30s.
• Docstring corrected to match the code.
• Note this narrows what the service validates. Revert both files together.

Graceful shutdown — 7 deployments

• 1200s / 6000s → 15s / 20s / 25s, sized just above each deployment's own request time.
• The machine is gone 120s after the notice regardless, so a 20-minute grace period only means the drain never terminates on its own.

Warm floors — config.py + 4 apps

• New _with_floor() applies min_replicas per deployment rather than on the shared presets, since AUTOSCALE_SPIKY_T2 is also used by batch-infer.
• nlp-chain, image-dag, cpu-fanout and long-runner get a floor of 2. A run with all four at zero took 29 cold-start 408s in its first 100 seconds, and the compressed ladder leaves far less room to survive that queue.
• Trade-off: min_replicas=0 was deliberate — this removes scale-from-zero coverage for those four.

Placement — anyscale_service.yaml

• Both worker groups go to PREFER_SPOT. Main sits on ON_DEMAND precisely because retirement did not fit the notice; everything above is what earns spot back, so these lines should never be reverted on their own.
• cpu-gpu-sim was held back at first, on the grounds that MuxModelWorker blocks readiness until all 20 models are pre-loaded and so recovers slowly. That does not hold up: _prewarm() gathers all 20 loads concurrently and each is a 10–40ms sleep over a 1–8 KiB bytearray. Measured mean replica startup over 7 days (n=477) is 30s for mux-model-worker vs 27s for mux-ingress, which loads no models at all — the preload is ~3s of a ~30s startup, and three deployments start slower.
• The exclusion was also where the money was. cpu-gpu-sim carries 3,014 of 3,226 measured worker node-hours/month (93%): simulated_gpu: 8 packs 8 replicas onto a node CPU alone would fill with 32, and those nodes absorb the CPU-only work too, leaving cpu-general at 7% of the fleet.

Latency buckets — anyscale_service.yaml

• 130000/150000 → 70000/90000. Those two bounds existed to stop a legitimate ~120s long-runner request interpolating P99 to ~282s and firing a false alert ~0.7×/day (RCA 2026-08-03). With the ceiling at 20s, nothing reaches them.

Important

The Grafana rule and the Notion acceptance table still carry long-runner's old 130s SLO. Against a 20s cap that is ~6× too high and the alert cannot fire until the rule's quantile boundary moves to 30000.

Verification

Five consecutive load tests at ~15,000 RPS:

Run	Duration	Requests	Failures	RPS	Result
10:14	915.0s	13,693,936	1	15,155.82	PASSED
10:35	935.7s	13,905,934	0	15,046.36	PASSED
11:11	940.1s	13,895,566	0	14,963.30	PASSED
11:49	914.8s	13,862,222	0	15,346.83	PASSED
12:12	936.8s	13,866,583	0	14,986.43	PASSED
total	78 min	69,224,241	1	~15,100	5 / 5

One failure in 69.2 million requests — 0.0000014%, against a 0.01% acceptance threshold. Cold-start timeouts went from 29 to zero.

What these runs do not show: no spot instance was reclaimed during any of the five, so they establish that the ladder is stable under sustained load and that the warm floors work. They do not yet demonstrate a live preemption being handled cleanly.

What this does not close

An old HAProxy worker can still accept one final request at any point up to its deadline, so the last 20s of a 30s worker life stay exposed. No value of hard-stop fixes that — raise it and the trailing window moves with it, lower it and the exposed fraction grows.

Closing it properly needs server-template plus runtime membership, so that changing the replica list never requires a reload and no old worker exists at all.