Skip to content

fix(branch): delete safekeeper WAL on branch delete, not just pageserver - #2

Merged
zxzinn merged 1 commit into
mainfrom
zxzinn/dev/bugfix/branch-delete-safekeeper-wal-orphan
Aug 17, 2026
Merged

fix(branch): delete safekeeper WAL on branch delete, not just pageserver#2
zxzinn merged 1 commit into
mainfrom
zxzinn/dev/bugfix/branch-delete-safekeeper-wal-orphan

Conversation

@zxzinn

@zxzinn zxzinn commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Summary

branch delete only called the pageserver's DELETE /timeline endpoint. Each safekeeper in the quorum independently persists a timeline's WAL and is never notified, so its on-disk directory becomes a permanent orphan on every delete.

Found this on the maiagent-dev-neon cluster: roughly 60 failed refresh-golden CI runs each left one orphaned timeline behind, filling all three safekeepers' 5Gi volumes to 100% and making every subsequent branch creation fail with No space left on device.

Changes

  • New internal/safekeeper package: a thin HTTP client mirroring internal/pageserver's style, calling the safekeeper's DELETE /v1/tenant/{tenant}/timeline/{timeline}.
  • branch delete now calls this on every safekeeper in the new --safekeeper-http-urls flag, after the pageserver delete succeeds.
  • Deliberately a separate flag from --safekeepers (which carries in-cluster host:5454 for compute's neon.safekeepers config, resolved pod-to-pod). The CLI process itself usually cannot resolve *.svc.cluster.local names, so reaching a safekeeper's HTTP API requires the caller to already have a path to it (e.g. one kubectl port-forward per safekeeper) and pass the resulting http://localhost:PORT URLs.
  • Optional and best-effort: an unset flag or a per-safekeeper failure is logged as a warning, not an error — the pageserver deletion already happened by that point and isn't reversible, so failing the command wouldn't undo anything, just hide the outcome.

Test plan

  • go build ./..., go vet ./..., go test ./... all pass
  • Manually verified against the live maiagent-dev-neon cluster: created a throwaway branch, deleted it with --safekeeper-http-urls pointing at three kubectl port-forwarded safekeepers, confirmed the timeline directory was actually removed from all three (kubectl exec ... ls /data/<tenant>/) — before the fix it was left behind on all three.
  • Confirmed the fix degrades gracefully: running branch delete without --safekeeper-http-urls still deletes the branch normally (matches pre-fix behavior for anyone not yet passing the new flag).

branch delete only called the pageserver's DELETE /timeline endpoint.
Each safekeeper in the quorum independently persists a timeline's WAL
and is never notified, so its on-disk directory becomes a permanent
orphan on every delete.

This filled all three safekeepers' 5Gi volumes to 100% on the
maiagent-dev-neon cluster after roughly 60 failed refresh-golden runs
each left one behind, which then made every new branch creation fail
with "No space left on device".

Adds an internal/safekeeper HTTP client and calls DELETE on each
safekeeper listed in the new --safekeeper-http-urls flag after the
pageserver delete succeeds. Kept as a separate flag from --safekeepers
(which carries in-cluster host:port for compute's neon.safekeepers
config) since the CLI process itself usually cannot resolve
*.svc.cluster.local names — reaching a safekeeper's HTTP API requires
the caller to already have a path to it (e.g. a port-forward) and pass
the resulting URL. The flag is optional and best-effort: a safekeeper
cleanup failure is logged as a warning but does not fail the command,
since the pageserver deletion already happened and is not reversible.
@zxzinn
zxzinn merged commit ca27462 into main Aug 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant