Skip to content

fix(wintermute): bound the repo backfill queue - #254

Open
afbase wants to merge 2 commits into
mainfrom
fix/wintermute-bound-repo-backfill-queue
Open

fix(wintermute): bound the repo backfill queue#254
afbase wants to merge 2 commits into
mainfrom
fix/wintermute-bound-repo-backfill-queue

Conversation

@afbase

@afbase afbase commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

The relay enumerator is a producer with no consumer whenever backfiller workers are disabled, which is the recommended setting where repo backfill is handled elsewhere. Nothing bounded it, so the queue grew for as long as the relay had repos to list.

Measured on a box consuming the live firehose with workers disabled: the queue reached 5.8 million entries in under five hours, and its Fjall partition 906 MB against 7 MB for every other partition combined. A heap profile over the same period showed the LSM read and compaction paths — LevelReader, segment::Range, MvccStream — holding roughly 3.5 GB of 4.8 GB peak live, allocating block buffers in proportion to the partition they were reading.

Resident memory tracked the queue across every restart cycle:

01:10   1,120,000 queued    4.24 GB
05:40   5,860,000           8.63 GB
06:40      20,000 restart   6.52 GB
11:40   1,640,000           8.53 GB

That reads as a leak — memory climbs, only a restart reclaims it — but it is a queue nothing drains. Peak live memory itself was stable across two consecutive six-hour windows (4.82 GB then 4.44 GB) while half a terabyte of allocation churned through, so nothing was being retained that should not have been.

Enumeration now stops at REPO_BACKFILL_MAX_QUEUE entries, default 250k. 0 restores the previous unbounded behaviour.

Two details worth review:

  • The bound is a parameter rather than read from the environment inside the loop, so the stopping behaviour is testable without a relay or a storage engine.
  • The queue length is tracked as enumeration proceeds rather than re-read. Fjall's len() scans the partition, so polling it per page would grow more expensive as the queue did. A concurrent drain only makes the tracked value an overestimate, which errs toward enqueueing less.

This does not change behaviour for anyone running backfiller workers unless their queue legitimately exceeds the bound, in which case enumeration pauses until it drains.

Test plan

  • cargo clippy -p rsky-wintermute --all-targets --no-deps -- -D warnings
  • cargo test -p rsky-wintermute against a database with the appview schema
  • cargo build --release -p rsky-wintermute
  • With BACKFILLER_WORKERS=0, ingester_repo_backfill_length stops at the bound instead of climbing
  • With workers enabled, the queue still drains and enumeration resumes
  • REPO_BACKFILL_MAX_QUEUE=0 reproduces the previous unbounded behaviour

The relay enumerator is a producer with no consumer whenever backfiller workers
are disabled, which is the recommended setting where repo backfill is handled
elsewhere. Nothing bounded it, so the queue grew for as long as the relay had
repos to list.

Measured on a profiling box consuming the live firehose with workers disabled:
the queue reached 5.8 million entries in under five hours and its Fjall
partition 906 MB, against 7 MB for every other partition combined. The LSM read
and compaction paths allocate block buffers in proportion, and those dominated
the heap profile. Resident memory tracked the queue on every cycle and only a
restart reclaimed it, which reads as a leak but is a queue nothing drains.

Enumeration now stops at REPO_BACKFILL_MAX_QUEUE entries, default 250k, with 0
meaning unbounded for anyone who wants the previous behaviour. The bound is
passed in rather than read from the environment inside the loop, so it can be
tested without a relay, and the queue length is tracked rather than re-read:
Fjall's len() scans the partition, so polling it per page would cost more the
larger the queue got.
…ig constants

The call site used a fully-qualified path while the file already had a
use crate::config::{...} block for every other constant it reads.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant