feat: add sliding sync (MSC4186) implementation#3673
Conversation
This commit adds native sliding sync support to Dendrite, implementing MSC4186 (Simplified Sliding Sync). Key features include: - New v4 sync endpoint at /_matrix/client/unstable/org.matrix.simplified_msc3575/sync - Sliding window room list with sorting and filtering - Room subscriptions for detailed room data - Extensions: to_device, e2ee, account_data, receipts, typing - Persistent connection state tracking - Support for lazy-loading room members - MSC3266 room summary endpoint - MSC4115 membership on events Also includes various fixes and improvements to existing sync functionality.
This commit includes several fixes for sliding sync receipt handling: - Clear receipt delivery state for specific rooms when timeline expansion is detected, ensuring receipts are re-delivered when the client resets its view of the room (initial:true with expanded_timeline:true) - Clear all connection receipts on fresh sliding sync connections - Copy-forward room configs to prevent perpetual limited:true state - Serialize m.read.private receipts under the correct type key - Extract invite_room_state from federated invite events properly - Populate timeline/state for NEVER rooms in incremental sync - Copy forward stream states for unchanged rooms in sliding sync These fixes address stuck unread badges in clients when rooms are entered/exited and timeline expansion occurs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This should fix element-hq#3669 We'd potentially send an empty string to `GenerateCreateContent`, even if we set the correct `roomVersion` before. ### Pull Request Checklist <!-- Please read https://matrix-org.github.io/dendrite/development/contributing before submitting your pull request --> * [x] I have added Go unit tests or [Complement integration tests](https://github.com/matrix-org/complement) for this PR _or_ I have justified why this PR doesn't need tests * [x] Pull request includes a [sign off below](https://element-hq.github.io/dendrite/development/contributing#sign-off) _or_ I have already signed off privately --------- Signed-off-by: Till Faelligen <2353100+S7evinK@users.noreply.github.com>
This is to complement the existing [Purge Room Admin API](https://element-hq.github.io/dendrite/administration/adminapi#post-_dendriteadminpurgeroomroomid) ### Pull Request Checklist <!-- Please read https://matrix-org.github.io/dendrite/development/contributing before submitting your pull request --> * [x] I have added Go unit tests or [Complement integration tests](https://github.com/matrix-org/complement) for this PR _or_ I have justified why this PR doesn't need tests * [x] Pull request includes a [sign off below](https://element-hq.github.io/dendrite/development/contributing#sign-off) _or_ I have already signed off privately --------- Signed-off-by: Till Faelligen <2353100+S7evinK@users.noreply.github.com>
Add GetLatestRoomConfigsBatch method to query multiple room configs in a single database query instead of N+1 queries. PostgreSQL uses DISTINCT ON for efficient single-query operation. SQLite falls back to iterating through rooms (acceptable for smaller deployments).
Refactor timeline expansion detection to use single batch query instead of N+1 pattern when processing room subscriptions.
Create single database snapshot for all filter operations in ApplyRoomFilters instead of creating N snapshots per room. Add WithSnapshot variants of getRoomName, isRoomEncrypted, and getRoomType to reuse the shared snapshot.
Add processListsAndCollectRooms and populateRoomDataForLists helper functions to eliminate ~120 lines of duplicated code between timeout and disconnect handlers. The helpers handle: - Processing room lists and collecting rooms that need data - Building room data for rooms in list operations
Add efficient delta operations for sliding sync room lists per MSC4186: - INSERT: Add room at specific index - DELETE: Remove room at specific index Implementation: - Add GenerateListOperations() to compute minimal operations - Detect insertions/deletions and order changes - Fall back to SYNC when changes exceed threshold (5 ops) - Update room tracking to handle INSERT operations This reduces bandwidth for incremental syncs when only a few rooms have changed position in the list.
Add spaces filtering support per MSC4186: - Query m.space.child state events from specified space rooms - Filter room list to include only direct children of those spaces - Child rooms are identified by state_key of m.space.child events - Only include children with valid "via" content (not removed) This allows clients to request rooms belonging to specific spaces using the "spaces" filter parameter.
|
|
|
ping @S7evinK |
|
What are the current blockers to line this PR up to be merged? |
|
@IngwiePhoenix This PR still requires a review from a Dendrite maintainer; however, not only is it enormous (152 changed files with over 20k added lines), but the project is currently in maintenance mode from Element's side and only receives security fixes. Reviewing this is likely not a priority for them. As an aside, @S7evinK no longer works at Element; he kindly puts in some of his spare time, but I can imagine that properly reviewing this PR would take more time than would be reasonable from that perspective. |
|
If people are interested in trying to get my work merged, a more reasonable approach would be trying to break it up into multiple, self-standing PRs. I agree that the current size of this PR is excessive, that's why I didn't submit one myself. A good starting approach would be to look at the individual branches on my fork; they follow the major features I tried to implement, but then everything got merged into the sliding-sync branch for integration and protocol testing. On top of that, the majority of the files that are touched are simply to bring error codes/messages more in line with spec (e.g. replacing 500 errors with more specific 40x errors) to improve compatibility with a few bots that are built with Synapse in mind. Unfortunately the original commit history contains some other proprietary work, so I had to release the sliding-sync branch as a flattened commit. This code also depends on my fork of gomatrixserverlib (again, mostly for error handling) so that's an obvious issue with merging. It will be a lot of work to do all this "properly"; it's obviously a lot easier to just let claude-code run wild making changes and then doing the protocol analysis to get it to work (Opus seemed particularly bad for this task). My workload has shifted away from this and I simply don't have time to manage that or maintain a separate fork. If merging this is too much for current @S7evinK, someone should step up and maintain a fork themselves. |
|
Until a decision is reached if and how to merge this PR, for anyone not in the #dendrite-dev:matrix.org room: A fork has been created a few days ago and a final name has been decided in the room. Dev discussions will continue in the #dendrite-dev:matrix.org room. |
I have successfully migrated just now to zendrite as described and it's working for me with Element X on Android and Element on Linux just fine. |
8 participants
This PR adds MSC4186, following the initial work by @jackmaninov from the discussion in #3236 (comment).
A image for testing is available at pats22/dendrite:sliding-sync.
I've used this image in an Dendrite instance of mine and got sliding sync to work, incl. Element X compatibility.
fix #3236
In addition to the work by @jackmaninov, several improvements to performance and maintainability were made:
Performance Improvements
Maintainability Improvements
Protocol Completeness
INSERT/DELETElist operations for efficient incremental updates (previously only SYNC was implemented)m.space.childstate events (previously returned an error)Technical Details
INSERT/DELETE Operations:
Spaces Filtering:
Pull Request Checklist
Signed-off-by:
Patrick Schratz <pat-s@mailbox.org>