Arguably a bit weird to be opening an issue about an MSC here but I'd argue MSC4186 Sliding Sync is so far along that it's quasi-specced. This concern also applies to several extension MSCs and so there's no 'natural' place to have a discussion about this, except maybe to make a self-contained issue like this one...
In recent MSCs proposing Sliding Sync extensions, there have been discussions about the pagination strategies of the items that those extensions will carry.
Different MSC extensions are going in different ways and it feels to me that we are missing some kind of 'stewardship' or 'taste' as to which direction the overall project should be going.
This issue seeks to lay some of the discussed options on the table and then perhaps we have a good start to think about the desired end state.
Strategies
Explicit backward pagination (using a companion endpoint)
In this strategy, the Sliding Sync response only includes a limited number of the latest items, e.g. 100 latest items. In other words, it includes items from the 'newest' side of the backlog.
The items are paginated by the request/response-level (global) pos token.
If the client missed some items, because there are more items than the limit allows to return in one response, then the response includes some token that the client can use to backpaginate the gap.
This backpagination occurs on a companion endpoint (not the /sync endpoint).
Extensions using this strategy:
Good:
- Clients get the latest data first, which is typically most useful.
- Clients don't have to retrieve the entire backlog if they don't want to.
Bad:
- Needs a separate companion endpoint.
- Clients need to be able to track multiple gaps, if they want to eventually get all the data.
- Clients need to (transactionally) persist another pagination token in their store.
- (For some types of item) races between subsequent
/sync requests and the companion endpoint backpagination requests mean that returned items need bump_stamps (or similar) to distinguish which response contains the latest information. (See: MSC4308: Thread Subscriptions extension to Sliding Sync § Use of bump_stamp)
Explicit forward pagination (using per-extension tokens)
In this strategy, the Sliding Sync extension does not use the request/response-level (global) pos token and instead specifies its own extension-level pagination tokens.
The extension response includes a limited number of items, from the 'oldest' side of the backlog and the extension-level pagination token advances.
Extensions using this strategy:
Good:
- No separate endpoint needed.
- Clients can turn off the extension when it's not desired and not lose their 'position'.
Bad:
- Clients get oldest data first, which is not always bad but often seems to be.
- Particularly of interest is that (for some extensions where this is relevant), the oldest data can then be 'overwritten' by newer data in a subsequent sync, briefly flashing stale data at the user even though the server had more recent data available. (In MSC4354 this risk has been referred to as 'Flickering' — TODO add a link to a discussion point)
- Clients need to (transactionally) persist another pagination token in their store.
Implicit Forward Pagination
In this strategy, the Sliding Sync response only includes a limited number of items, from the 'oldest' side of the backlog.
Pagination is controlled by the request/response-level (global) pos token.
Subsequent sliding sync requests will walk forwards through the backlog of items
No extensions in Sliding Sync use this strategy. However, some 'oldschool' sync streams use this strategy (e.g. to-device messages).
Good:
- Clients don't need to manage any extra tokens, all the pagination happens 'automatically', which is easy for the clients to deal with.
Bad:
- Clients get oldest data first, which is not always bad but often seems to be.
- The
/sync request will 'hot-loop' (return immediately) until the backlog is cleared. This is intensive for mobile clients and also might starve out the processing of other streams
- Clients can't control/slow down receiving the entire backlog.
- This strategy is one of the aspects that some consider to have made oldschool sync complex and brittle. ([ref])
No pagination
Technically, we can choose to not paginate at all and just return the entire set of unseen data in the Sliding Sync response.
Good:
Bad:
- Sends data of unbounded size to the client
Extensions using this strategy:
Arguably a bit weird to be opening an issue about an MSC here but I'd argue MSC4186 Sliding Sync is so far along that it's quasi-specced. This concern also applies to several extension MSCs and so there's no 'natural' place to have a discussion about this, except maybe to make a self-contained issue like this one...
In recent MSCs proposing Sliding Sync extensions, there have been discussions about the pagination strategies of the items that those extensions will carry.
Different MSC extensions are going in different ways and it feels to me that we are missing some kind of 'stewardship' or 'taste' as to which direction the overall project should be going.
This issue seeks to lay some of the discussed options on the table and then perhaps we have a good start to think about the desired end state.
Strategies
Explicit backward pagination (using a companion endpoint)
In this strategy, the Sliding Sync response only includes a limited number of the latest items, e.g. 100 latest items. In other words, it includes items from the 'newest' side of the backlog.
The items are paginated by the request/response-level (global)
postoken.If the client missed some items, because there are more items than the limit allows to return in one response, then the response includes some token that the client can use to backpaginate the gap.
This backpagination occurs on a companion endpoint (not the
/syncendpoint).Extensions using this strategy:
Good:
Bad:
/syncrequests and the companion endpoint backpagination requests mean that returned items needbump_stamps (or similar) to distinguish which response contains the latest information. (See: MSC4308: Thread Subscriptions extension to Sliding Sync § Use ofbump_stamp)Explicit forward pagination (using per-extension tokens)
In this strategy, the Sliding Sync extension does not use the request/response-level (global)
postoken and instead specifies its own extension-level pagination tokens.The extension response includes a limited number of items, from the 'oldest' side of the backlog and the extension-level pagination token advances.
Extensions using this strategy:
Good:
Bad:
Implicit Forward Pagination
In this strategy, the Sliding Sync response only includes a limited number of items, from the 'oldest' side of the backlog.
Pagination is controlled by the request/response-level (global)
postoken.Subsequent sliding sync requests will walk forwards through the backlog of items
No extensions in Sliding Sync use this strategy. However, some 'oldschool' sync streams use this strategy (e.g. to-device messages).
Good:
Bad:
/syncrequest will 'hot-loop' (return immediately) until the backlog is cleared. This is intensive for mobile clients and also might starve out the processing of other streamsNo pagination
Technically, we can choose to not paginate at all and just return the entire set of unseen data in the Sliding Sync response.
Good:
Bad:
Extensions using this strategy: