Package
@atcute/jetstream 2.0.1
Problem
JetstreamSubscription delivers WebSocket messages through
@mary-ext/event-iterator, whose queue is observable but unbounded. A sequential
consumer that awaits durable I/O can therefore accumulate complete Jetstream
records faster than it drains them.
Closing the iterator with return() is not a safe pressure mechanism because it
clears the queue. Reconnecting from the application's durable processed cursor
then replays the cleared range; under sustained pressure this can cycle without
catching up. Fresh subscriptions using a URL array also intentionally roll back
ten seconds, which amplifies that starvation mode.
Production observations
In a Cloudflare Durable Object consumer:
- Unbounded intake reached roughly 177–185 MiB memory P90.
- Polling
iterator.queued, calling return(), and reconnecting reduced memory
but caused discontinuous readiness and cursor starvation.
- A queue-preserving prototype that closes only socket intake at a high-water
mark, drains the existing iterator, and resumes the same endpoint at a
low-water mark preserved cursor continuity.
- With 200/50 high/low watermarks, memory P90 fell to about 30 MiB. In an early
repeated sample it processed 5,750 posts with 29 pause/28 resume cycles, no
reconnect or run errors, and the queue never exceeded 200.
Suggested API
Would you consider first-class bounded buffering on JetstreamSubscription?
new JetstreamSubscription({
url,
maxQueuedEvents: 200,
resumeQueuedEvents: 50,
onBufferPause(stats) {},
onBufferResume(stats) {},
})
Desired semantics:
- Stop or close WebSocket intake at the high-water mark without clearing
already admitted iterator events.
- Continue ordered draining.
- Resume the same endpoint at the newest admitted cursor when the low-water
mark is reached, avoiding cross-endpoint rollback for a deliberate pressure
pause.
- Preserve normal multi-endpoint rollback for actual failover.
- Expose current/max queued events, oldest queued age, pressure state, and
pause/resume counts.
- On consumer termination, retain the current
return() behavior that clears
the queue and closes the socket.
I can prepare a PR based on the validated prototype if this API direction fits
the project.
Package
@atcute/jetstream2.0.1Problem
JetstreamSubscriptiondelivers WebSocket messages through@mary-ext/event-iterator, whose queue is observable but unbounded. A sequentialconsumer that awaits durable I/O can therefore accumulate complete Jetstream
records faster than it drains them.
Closing the iterator with
return()is not a safe pressure mechanism because itclears the queue. Reconnecting from the application's durable processed cursor
then replays the cleared range; under sustained pressure this can cycle without
catching up. Fresh subscriptions using a URL array also intentionally roll back
ten seconds, which amplifies that starvation mode.
Production observations
In a Cloudflare Durable Object consumer:
iterator.queued, callingreturn(), and reconnecting reduced memorybut caused discontinuous readiness and cursor starvation.
mark, drains the existing iterator, and resumes the same endpoint at a
low-water mark preserved cursor continuity.
repeated sample it processed 5,750 posts with 29 pause/28 resume cycles, no
reconnect or run errors, and the queue never exceeded 200.
Suggested API
Would you consider first-class bounded buffering on
JetstreamSubscription?Desired semantics:
already admitted iterator events.
mark is reached, avoiding cross-endpoint rollback for a deliberate pressure
pause.
pause/resume counts.
return()behavior that clearsthe queue and closes the socket.
I can prepare a PR based on the validated prototype if this API direction fits
the project.