Summary
v0.2.18's stream quarantine (stream.quarantine, unquarantine, …) is the right primitive for "move this bad batch of updates aside" — we use it heavily. But operationally we also need the other control: stop workers from compacting this room until an operator resumes it, while the room stays writable for clients.
Without it, the failure loop we hit in production is:
- A room repeatedly fails compaction (whatever the cause).
- We quarantine the current stream — necessary, but not sufficient.
- Clients keep writing; the live stream refills.
- Workers pick the room up again, hit the same condition, fail again.
- Repeat — the worker queue fills with
auto-quarantined error tasks for the same room.
We built this downstream as a Redis marker ({prefix}:document_quarantine:{org}:{docid}:{branch} + an index set) that our worker consults before compacting, our admin API reports, and our recovery runbook drives. It works, but it is glue around y/hub internals and has to be re-validated on every upgrade.
Ask
A room-level worker-pause primitive, e.g.:
await yhub.quarantineRoomForWorkers(room, { reason, details })
await yhub.isRoomQuarantinedForWorkers(room)
await yhub.unquarantineRoomForWorkers(room)Expected behavior: websocket and REST writes unaffected; worker compact tasks for the room acked/skipped without processing; explicit operator-driven resume; composes with (does not replace) stream quarantine. Related detail from our incident history: repeated auto-quarantines leave one nop-marker quarantine stream behind per attempt — 120 of them on one document — so a backlog inspection that counts raw XLEN reports phantom pending messages forever; a first-class primitive could also own "empty quarantine stream" hygiene.
Summary
v0.2.18's stream quarantine (
stream.quarantine,unquarantine, …) is the right primitive for "move this bad batch of updates aside" — we use it heavily. But operationally we also need the other control: stop workers from compacting this room until an operator resumes it, while the room stays writable for clients.Without it, the failure loop we hit in production is:
auto-quarantinederror tasks for the same room.We built this downstream as a Redis marker (
{prefix}:document_quarantine:{org}:{docid}:{branch}+ an index set) that our worker consults before compacting, our admin API reports, and our recovery runbook drives. It works, but it is glue around y/hub internals and has to be re-validated on every upgrade.Ask
A room-level worker-pause primitive, e.g.:
Expected behavior: websocket and REST writes unaffected; worker compact tasks for the room acked/skipped without processing; explicit operator-driven resume; composes with (does not replace) stream quarantine. Related detail from our incident history: repeated auto-quarantines leave one nop-marker quarantine stream behind per attempt — 120 of them on one document — so a backlog inspection that counts raw XLEN reports phantom pending messages forever; a first-class primitive could also own "empty quarantine stream" hygiene.