Enforce pagination of resource listings from the server-side

[soxofaan]

Context: we're battling performance issues caused by job listings from user with a lot of jobs, so we're looking into implementing/improving pagination of job listings.

Some notes about what the spec currently states about pagination of job listings:

GET /logs
Lists all batch jobs submitted by a user
...
query param limit:
This parameter enables pagination
Pagination is OPTIONAL: back-ends or clients may not support it. Therefore it MUST be implemented in a way that clients not supporting pagination get all resources regardless. Back-ends not supporting pagination MUST return all resources

In nutshell: pagination is optional and should be opt-in from client side (by setting limit param).

I'm afraid however (as we currently see in practice) that this is not a sustainable behavior: the backend must also be able to limit the job listing and enable pagination. It's not uncommon for larger use cases to have users with multiple thousands of jobs, and fetching all the related metadata/status of all these jobs regularly is quite expensive, especially as most users just look at, say the last 10 jobs anyway.

Some more concrete questions:

• is there any wiggle room to finetune the spec to also allow the backend to enable pagination in some way?
• what should a backend do, under existing spec, if they want to enable pagination even if user/client didn't use limit? There doesn't seem like a solution that is both pragmatic and follows the spec to the letter.
  • E.g. can a backend use a default limit value, e.g. 100, which is different from "unset by default" in the spec?
  • Raise an error if user has too much jobs

[m-mohr]

Unfortunately, it seems this is only possible in v2.0 in the spec. What we can do in the meantime is to implement pagination in clients so that if a back-end supports it, it is automatically used.

I see issues in Web Editor and Python Client, but we should probably open issues in JS and R (and Julia if they have job listings implemented). Opened one for JS: Open-EO/openeo-js-client#64

[m-mohr]

We discussed potential default values for the clients and thought 100 or maybe 50 seem to be reasonable defaults.

[soxofaan]

Another pagination thing that might deserve some clarification in current spec is how jobs are partitioned in pages.
I guess it makes most sense to order by creation date from recent to older, so that first page (the initial listing request) has the most recently created jobs

[m-mohr]

That's up to the backend to decide, but I guess it makes most sense for backends to sort by creation or last update date. I guess we can add a recommendation, but I'm not sure what the best recommendation is :-)

[soxofaan]

Usability- and user-experience-wise I would require to show most recent jobs first, those are the jobs users are most interested in in practice, so you don't want them to have to wait for possibly tens of HTTP roundtrips just to get the most interesting page

[m-mohr]

Yeah, but most recent for me would mean sorted by "updated".
But updated is optional, we only require "created".

[soxofaan]

with recent I mean "by created" here (not "updated"), indeed because it is a required value

[m-mohr]

created is probably also better because it's stable so that entries don't jump between pages... okay, convinced ;-)
But then, what's the recommended value for the other paginated resources?