Summary
y/hub currently emits no operational events or metrics from the server, so production triage (why did this open take 30s? which client is dropping on backpressure? which rooms drive fan-out volume?) requires fleet-wide debug logging. We patched in a server.events config and it now powers 100% of our y/hub observability — ~220M events/week feed our dashboards from it. Offering it upstream:
server: {
events: {
wsUpgrade?: (e: { room?, url?, userId?, accessType?, status: 'success'|'error'|'aborted', durationMs, errorType?, error?, httpStatus? }) => void | Promise<void>,
documentOpen?: (e: { room, userId?, clientId?, gc?, status, durationMs, readDurationMs?, stateVectorDurationMs?, docSize?, syncStep1Bytes?, syncStep2Bytes?, awarenessBytes?, bytesOut?, lastClock?, errorType?, error? }) => ...,
fanout?: (e: { room, userId?, operation: 'fanout_ydoc_update'|'fanout_awareness_update', status, durationMs, bytesOut?, itemCount?, errorType?, error? }) => ...,
wsClose?: (e: { room, userId?, clientId?, status, durationMs, closeCode, closeReason? }) => ...,
}
}Handlers are invoked inside try/catch (a broken handler can never affect the connection), async rejections are swallowed with a log. Two details that proved especially valuable:
- Backpressure visibility.
sendData's uWS result is surfaced: a dropped fan-out or initial-sync frame emits errorType: 'backpressure'. Today upstream drops these silently — the client desyncs with zero server-side signal.
taskComplete.outcome: 'persisted' | 'trim-only' | 'unknown' on the worker event, so consumers can distinguish "durable state advanced" (drive webhooks/search indexing) from a trim-only pass. In v0.7.0 that needs setting at both trim-only exits (the pre-check and the new post-merge duplicate-clock recheck) plus the persisted exit.
Sample numbers this instrumentation surfaced for us in one production week: 95k document opens averaging 2.24 MB each (213 GB of initial-sync egress), awareness at 89% of all operations, and open p95 dominated by the read+merge phase — none of which is visible without hooks. We run this via a package patch and are happy to send a PR.
Summary
y/hub currently emits no operational events or metrics from the server, so production triage (why did this open take 30s? which client is dropping on backpressure? which rooms drive fan-out volume?) requires fleet-wide debug logging. We patched in a
server.eventsconfig and it now powers 100% of our y/hub observability — ~220M events/week feed our dashboards from it. Offering it upstream:Handlers are invoked inside try/catch (a broken handler can never affect the connection), async rejections are swallowed with a log. Two details that proved especially valuable:
sendData's uWS result is surfaced: a dropped fan-out or initial-sync frame emitserrorType: 'backpressure'. Today upstream drops these silently — the client desyncs with zero server-side signal.taskComplete.outcome: 'persisted' | 'trim-only' | 'unknown'on the worker event, so consumers can distinguish "durable state advanced" (drive webhooks/search indexing) from a trim-only pass. In v0.7.0 that needs setting at both trim-only exits (the pre-check and the new post-merge duplicate-clock recheck) plus the persisted exit.Sample numbers this instrumentation surfaced for us in one production week: 95k document opens averaging 2.24 MB each (213 GB of initial-sync egress), awareness at 89% of all operations, and open p95 dominated by the read+merge phase — none of which is visible without hooks. We run this via a package patch and are happy to send a PR.