Non-atomic server disable via Runtime API briefly leaves a server routable at 127.0.0.1 to 503 (SC) under pod churn / deployments

[apietroniront]

When a backend server slot is disabled (a pod leaves: scale-down, rollout, restart), the
controller updates it through the Runtime API by sending, in this order:

set server <backend>/<srv> addr 127.0.0.1 port 1
set server <backend>/<srv> state maint

Because these are two independent Runtime API commands, there is a brief window between
them in which the server is still in its previous ready state but its address has
already been changed to 127.0.0.1. During that window the load balancer still considers
the server up and can route a request to it, which then connects to the loopback placeholder
and is refused. With retries (default 3) and without option redispatch, all retries hit
the same dead slot → the client gets a 503.

This surfaces as sporadic 503s under pod churn, and as spikes during deployments (a rolling
update disables many slots at once). Graceful shutdown (preStop/terminationGracePeriodSeconds)
does not help, because the slot is pointed at the loopback, not at the draining pod.

Environment

• Controller: v3.1.13 (image haproxytech/haproxy-alpine:3.1, client-native v5.1.18).
• Also reported on v3.1.14 see [question] ingress serving 503 on deployment restarts #714.
• Backends use a config where each server has low maxconn and there is heavy pod churn
  (autoscaling by queue), which makes the window observable at scale. No backend health check
  (check) is required to reproduce.

What happens (code references)

The live (reloadless) apply path is SyncBackendSrvs in pkg/haproxy/api/runtime.go, which
builds the runtime update per direction:

• disable (srv.Address == "") → {IP: "127.0.0.1", Port: 1, State: "maint"}
• enable → {IP: <pod-ip>, State: "ready"}

SetServerAddrAndState then serializes this as set server … addr <IP> …;set server … state <STATE>; addr is always emitted before state, for both directions. This ordering is
safe for enable (the in-between state is maint + real IP → not routable) but unsafe for
disable (the in-between state is ready + 127.0.0.1 → routable).

On the HAProxy core side, the two ;-separated commands are processed independently
(src/cli.c splits on ;), and each set server sub-command takes and releases the server
lock on its own (cli_parse_set_server in src/server.c): the addr branch calls
srv_update_addr_port without touching the admin/operational state, and the state maint
branch runs afterwards as a separate locked operation. So between the two commands the lock is
released and other worker threads can observe/route to the server as ready + 127.0.0.1.

Log signature

… <backend>/SRV_k 127.0.0.1:1 …/…/-1/…/… [503] … SC-- …/…/…/…/3 …
                  ^^^^^^^^^^^^                        ^^      ^
                  server addr = loopback              SC      rc=3 (all retries same slot)

(%si=127.0.0.1, termination state SC = server connection failure, Tc=-1 = connect failed,
%rc=3 = 3 retries, no redispatch.)

How to reproduce

Deterministic (isolates the exact window): on a backend with an active server, drive steady
traffic and, via the Runtime API, set the server's address to the loopback placeholder without
putting it in maintenance:

set server <backend>/<srv> addr 127.0.0.1 port 1

Requests balanced to that server now return 503 with the signature above (SC, rc=3). Restore
with set server <backend>/<srv> addr <pod-ip> port <port>. This holds the transient state open
on purpose; in real traffic it lasts only the sub-millisecond gap between the two commands.

Natural: trigger pod churn (scale-down or a rolling deployment with high surge) under load —
503s appear correlated with the disable events.

Impact

• Sporadic client-facing 503s under pod churn; spikes during deployments (rollout = burst of
  slot disables), even with graceful shutdown configured.
• option redispatch mitigates it (the retry is re-dispatched to a healthy server), but the
  coverage is partial under a burst when many slots are transitioning at once and the
  re-dispatch can re-pick a transitioning slot — consistent with [question] ingress serving 503 on deployment restarts #714, where redispatch was set
  and 503s still occurred on deployment restarts.