Skip to content

Conversation

@turt2live
Copy link
Member

@turt2live turt2live commented Dec 16, 2025

Note: This cleans up previous code introduced by #19268

When first writing the API, "unquarantining" media was not properly considered. If media was unquarantined and an application was treating from tokens as long-lived when listing quarantined media, the endpoint could skip rows like so:

  1. Media A is quarantined
  2. Media B is quarantined
  3. Application lists quarantined media, caches from=2
  4. Media A is unquarantined, shifting B from row 1 to 0
  5. Media C is quarantined, becoming the new row 1
  6. Media D is quarantined
  7. Application lists media with from=2, gets [D]

To fix this, we invent a pagination token which uses time and a relative index. It's not super stable still because the relative index can still change, but it's likely stable enough for most usage (iterate as fast as possible to the end).

If an application requires a proper time-based stable token, it can generate a timestamp then append -0 to it to set the relative position to the zeroth row. This may return rows the application has already seen, as described by the admin API docs. This particular method of generating the timestamp manually is not documented because it's not as stable as relying on the last seen next_batch's internal timestamp.

This PR doesn't shift the whole endpoint to timestamp-only tokens because the prior PR populates rows with 0 for a timestamp, which may span thousands (or millions) of rows, breaking the ability to use limit properly.

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
    • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
  • Code style is correct (run the linters)

@turt2live turt2live changed the title Travis/fix quarantine list Fix "list quarantined media" API to have semi-stable ordering Dec 16, 2025
Comment on lines +87 to +94
`from` and `limit` are optional parameters, and default to the first page and `100` respectively. `from` is the `next_batch`
token returned by a previous request and `limit` is the number of rows to return. Note that `next_batch` is not intended
to survive longer than about a minute and may produce inconsistent results if used after that time. Neither `from` or
`limit` is a timestamp, though `from` does encode a timestamp.

If you require a long-lived `from` token, split `next_batch` on `-` and combine the first part with a `0`, separated by
a `-` again. For example: `1234-5678` becomes `1234-0`. Your application will need to deduplicate `media` rows it has
already seen if using this method.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do accept that this is custom, weird, and not-quite how pagination works. My defence is I have a custom, weird, not-quite-paginating use case 😇

(though, if someone more qualified wants to make this use PaginationHandler instead, I'd happily close this PR in favour of that one)

Comment on lines +333 to +334
elif len(start) > 0:
start_index = int(start)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably don't need this backwards compatibility given the PR which introduced the endpoint has only been on develop for a few days as of writing. I've kept it anyway because I honestly just don't want to fight the unit tests too much. I can be convinced to enter battle, however.

@turt2live turt2live force-pushed the travis/fix-quarantine-list branch from c7ec79d to 1128c59 Compare December 16, 2025 06:28
@turt2live turt2live force-pushed the travis/fix-quarantine-list branch from 1128c59 to 516b740 Compare December 16, 2025 06:41
@turt2live turt2live marked this pull request as ready for review December 16, 2025 06:53
@turt2live turt2live requested a review from a team as a code owner December 16, 2025 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants