Skip to content

unquarantine loads the whole quarantine stream into memory and replays it in one MULTI — OOMs on multi-GB quarantines #66

Description

@bgeils

Summary

Stream.unquarantine XRANGEs the entire quarantine stream into process memory and replays it as one atomic MULTI (src/stream.js:502-524 in v0.7.0). Quarantines can hold weeks of updates — we have had multi-GB quarantine streams in production — so restoring one either OOMs the server process or stalls Redis for the duration of a single giant EXEC.

Suggested fix (running in production, offered as a PR)

Read and re-inject in bounded batches (we use 1,000 entries / 8 MB per EXEC), deleting the quarantine key only after the final batch:

  • memory and per-EXEC work become O(batch);
  • a crash between batches leaves the quarantine key intact and some entries duplicated in the live stream — safe, because Yjs updates and awareness frames are idempotent under re-application, so the operation is simply re-runnable;
  • v0.7.0's auth:check:v1 drop-on-replay filter is preserved inside the loop.

Return value note: ours returns the count of entries walked (matching current behavior) rather than entries re-injected.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions