Title: Branch push verification memory and time scale with push size (one store lookup per fragment, unbounded concurrency)
Summary
Pushing a large revision makes loreserver's memory, CPU and BranchPush handler time grow with the number of fragments in the push. On a small S3/DynamoDB-backed server a 20 GB push (~80k fragments) OOM-kills loreserver or fails with Request handler timeout exceeded.
Where
verify_fragments (lore-server/src/grpc/handlers/branch_push.rs) → state::collect_new_fragments → collect_new_addresses (lore-revision/src/state.rs). The latter spawns a task per fragment address calling immutable_store().get_metadata(); on the AWS store each call is a DynamoDB get_item plus an S3 HEAD. MAX_TASKS = 1000 bounds only one JoinSet, and the recursion nests one set per fragmented file, so the effective concurrency multiplies. verify_fragments then spawns every 100-address query batch at once with no in-flight cap.
Evidence
- Production (t3a.medium, 4 GB, no swap): 20 GB push → anonymous heap ~3.4 GB, kernel OOM kill ×3 in a restart loop; on a newer build, survives but every push fails with the 50 s handler timeout at 200 % CPU.
- Reproduction on an identical host with a 5 GB synthetic push: content transfer flat at ~240 MB RSS;
lore push (BranchPush) 250 → 909 MB in 40 s; a second run hit the handler timeout.
- Allocation profile of the push window (
LORE_ALLOCATOR=tracking, symbolized): live growth is AWS SDK request state — hyper read buffers, TLS connector churn, ~148k header maps, ~70k SigV4 signings in 70 s — plus collect_new_addresses task state. About one AWS request per fragment, thousands in flight.
- Retries balloon immediately because the content is already stored, so the push goes straight to verification.
Fix
PR (follow-up link) bounds the fan-out: a shared semaphore for the per-address lookups across the recursion, and an in-flight cap on query batches. With it, a 1 GB push adds ~80 MB and completes in 9 s; the 20 GB production push completed with BranchPush peaking under 800 MB.
Follow-on to consider
Batch the existence checks through ImmutableStore::query (100 per request) instead of per-address get_metadata — ~100× fewer requests. StoreMatchResult lacks the PayloadFragmented flag, so the tree walk would need to know which addresses are fragment lists (including nested lists for very large files).
Title: Branch push verification memory and time scale with push size (one store lookup per fragment, unbounded concurrency)
Summary
Pushing a large revision makes loreserver's memory, CPU and BranchPush handler time grow with the number of fragments in the push. On a small S3/DynamoDB-backed server a 20 GB push (~80k fragments) OOM-kills loreserver or fails with
Request handler timeout exceeded.Where
verify_fragments(lore-server/src/grpc/handlers/branch_push.rs) →state::collect_new_fragments→collect_new_addresses(lore-revision/src/state.rs). The latter spawns a task per fragment address callingimmutable_store().get_metadata(); on the AWS store each call is a DynamoDBget_itemplus an S3HEAD.MAX_TASKS = 1000bounds only oneJoinSet, and the recursion nests one set per fragmented file, so the effective concurrency multiplies.verify_fragmentsthen spawns every 100-address query batch at once with no in-flight cap.Evidence
lore push(BranchPush) 250 → 909 MB in 40 s; a second run hit the handler timeout.LORE_ALLOCATOR=tracking, symbolized): live growth is AWS SDK request state — hyper read buffers, TLS connector churn, ~148k header maps, ~70k SigV4 signings in 70 s — pluscollect_new_addressestask state. About one AWS request per fragment, thousands in flight.Fix
PR (follow-up link) bounds the fan-out: a shared semaphore for the per-address lookups across the recursion, and an in-flight cap on query batches. With it, a 1 GB push adds ~80 MB and completes in 9 s; the 20 GB production push completed with BranchPush peaking under 800 MB.
Follow-on to consider
Batch the existence checks through
ImmutableStore::query(100 per request) instead of per-addressget_metadata— ~100× fewer requests.StoreMatchResultlacks thePayloadFragmentedflag, so the tree walk would need to know which addresses are fragment lists (including nested lists for very large files).