Skip to content

Fix KetamaHashing panic on zero-weight backend killing load balancer background task - #1007

Open
sinhaparth5 wants to merge 1 commit into
cloudflare:mainfrom
sinhaparth5:fix/ketama-zero-weight-panic
Open

Fix KetamaHashing panic on zero-weight backend killing load balancer background task#1007
sinhaparth5 wants to merge 1 commit into
cloudflare:mainfrom
sinhaparth5:fix/ketama-zero-weight-panic

Conversation

@sinhaparth5

Copy link
Copy Markdown

Fixes #1006.

Root cause

KetamaHashing::build_with_config passes Backend::weight (an unvalidated usize) straight into pingora_ketama::Bucket::new, which asserts weight != 0 and panics otherwise. A ServiceDiscovery implementation that uses weight 0 as a draining marker, or a config/DNS typo, is enough to trigger it.

That call runs inside LoadBalancer::update(), called from the background update loop in background.rs with no catch_unwind or spawn_blocking around it. The newer LoadBalancerGroup rebuild path already guards the equivalent call with spawn_blocking and JoinError::is_panic(), but the plain LoadBalancer<S>::run() path does not. Once the panic fires, the update task ends and is never restarted: backend membership and health check state freeze permanently (or the whole process aborts, under panic = "abort"), from a single zero-weight entry in one discovery response.

Fix

Clamp the weight to a minimum of 1 at the point it crosses from Backend into Bucket, in KetamaHashing::build_with_config. This is the only place backend weights feed into pingora-ketama in this crate, so it is the right boundary to treat the value as untrusted input.

Testing

  • Added test_ketama_zero_weight_backend_does_not_panic, which builds a KetamaHashing ring with a zero-weight backend and checks it is still reachable.
  • Reproduced the panic on an isolated GCE VM (torn down afterward): built the parent commit with a #[should_panic] version of the same test and confirmed it panics with weight must be at least one; the fixed commit does not panic and the same test passes.
  • cargo test -p pingora-load-balancing passes in full (79 tests) on the fixed commit.

KetamaHashing::build_with_config passes Backend::weight straight into
pingora_ketama::Bucket::new, which asserts weight != 0 and panics
otherwise. Backend::weight is an unvalidated usize set by whatever
ServiceDiscovery is in use (some treat 0 as a draining/disabled
marker), so a single zero-weight backend from discovery reaches that
assert.

The panic happens inside LoadBalancer::update(), called from the
background update loop in background.rs with no catch_unwind or
spawn_blocking around it (unlike the newer LoadBalancerGroup rebuild
path, which already guards against selector-build panics). The
panicking task ends and is never restarted, so backend membership and
health check state freeze permanently after the first such discovery
response.

Clamp the weight to a minimum of 1 at the point it crosses from
Backend (untrusted, unvalidated) into Bucket (which requires a
positive weight), instead of letting it panic.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Zero-weight backend panics KetamaHashing and kills the load balancer's background update task

1 participant