fix: use ReliableClusterSend for large run events - #2323
Open
calebroseland wants to merge 4 commits into
Open
Conversation
Playbook run websocket events (playbook_run_updated, playbook_run_updated_incremental, playbook_run_created) serialize full or partial PlaybookRun payloads that can exceed the ~49KB cap of the best-effort UDP cluster path, causing events to be silently dropped for large runs. Route these large, essential events through new reliable poster variants that set WebsocketBroadcast.ReliableClusterSend, so they are delivered over the reliable TCP-backed cluster channel. Small team-scoped events (playbook created/archived/restored) and the global settings-changed event keep the default best-effort path; their payloads are well under the limit. Context: /longshot MM-69386 — large/essential WS events sent reliably.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds reliable websocket publish methods to the Bot API, updates mocks and tests, and switches playbook and configuration websocket event broadcasts to the reliable variants. ChangesReliable WebSocket Publishing Across Playbook Events
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
JulienTant
approved these changes
Jun 23, 2026
Member
|
/update-branch |
Contributor
|
I do wonder if we should also use the reliable transport for |
Following up on review feedback: playbook_created, playbook_archived, playbook_restored and settings_changed were left on the best-effort UDP cluster path. Their payloads are tiny, but best-effort can drop packets outright in HA deployments, leaving connected clients with a stale playbook list or missed settings change. These events are rare and essential, so deliver them over the reliable TCP-backed channel too. Add PublishWebsocketEventToTeamReliable and PublishWebsocketEventGlobalReliable to complete the reliable poster set, and document the criterion (essential + low-frequency) so future high-frequency events stay on best-effort and don't saturate the shared reliable channel. Context: /longshot MM-69386 — PR #2323 review (hanzei): use reliable transport for the additional rare/essential events, not just large ones.
# Conflicts: # server/config/service.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Playbook run websocket events serialize full or partial
PlaybookRunpayloads. Whenmodel.WebsocketBroadcast.ReliableClusterSendis left at its default (false), inter-node delivery uses the best-effort UDP path, which (a) caps the marshalled event at ~49077 bytes and (b) may drop events. Large runs exceed the cap, soplaybook_run_updatedhas been silently dropped for customers and internally.This routes the large, essential run events through new reliable poster variants that set
ReliableClusterSend: true, delivering them over the reliable, TCP-backed cluster channel.Events switched to reliable (carry full/partial
PlaybookRundata):playbook_run_updated(channel + per-user)playbook_run_updated_incremental(channel + per-user) —ChangedFieldscan include full checklist/timeline/status sub-treesplaybook_run_created(both publish sites)Left on best-effort (payloads well under the limit):
playbook_created/playbook_archived/playbook_restored({"teamID": id}) and the globalsettings_changedevent.Implementation
server/bot/poster.go: extracted a sharedpublishWebsocketEventhelper (no behavior change to existing methods) and addedPublishWebsocketEventToChannelReliable/PublishWebsocketEventToUserReliable.server/bot/bot.go: added both methods to thePosterinterface; regeneratedmock_poster.go.server/bot/poster_test.gocovering broadcast scope, theReliableClusterSendflag per method, and the shared payload marshalling path.Ticket Link
https://mattermost.atlassian.net/browse/MM-69386
Checklist