Pagination the REST way : 'Content-Range' header

[Nayte91]

Hello, I move this post from discussions (#2804) to RFC issue, because I'll try to implement it,

I was lurking on the internet when I wondering how to implement easily and in a generic way a count for a given resource, the REST way?

For example if you have a dictionary API and you already have a 'count' word definition, it's unconvenient to add a api.foo/words/count route. It's also not very RESTful to create and impose the usage of a new verb COUNT, dedicated for this operation.

What we want, ideally, is to have the number of items in the pagination of this current request, ('30 items' in your book demo API), how many resources can be found on every page ('100 items' in total?), and therefor you can calculate how much pages there is. Also, for economic purpose, we want the content of the response to be empty or totally different of the other call on this resource, for example with just those counts.

And I feel like there is a very nice way to deal with this: as a user, you can request your book/ resource with the HEAD verb, that will ensure no body in response, and as an API provider, you can enrich your response with this nice 'Content-Range' header!

HEAD verb

HEAD works like... HEAD, it's already handled, so nothing to say more.

Content-Range HTTP header

'Content-Range' header is intended to carry a bunch of very clever information, as the common example is:

Content-Range: bytes 200-1000/67589

Where we can find the unit, the current range, and / the total count. Implicitely, you can find the number of items served in this response, here 1000-200 = 800 bytes, the current page, and the total number of pages. The tricky part is that the <unit> is usually "bytes", but the specifications doesn't limit!

So there's a scenario where API-Platform can answer a Content-Range: books 1-30/201 when requesting 'https://demo.api-platform.com/books':

• The <unit> is the resource on plural, an information API-Platform already has when naming routes,
• The <range-start>-<range-end> is defined by pagination,
• The <size> is the total count of this request.

I feel like there is some MASSIVE advantages in it:

• very convenient info about current pagination, even in regular JSON answers (as hydra returns a hydra:totalItems key for it),
• clever usage of HEAD verb for economic requests as you don't need to charge all the jsonLD payload to have info,
• native HTTP solution that totally adheres in REST,
• saves few headaches of 'how will provide a count for this request on my API?' with an out of the box solution; JSON+LD advantages for everyone!

You can argue that if specifications don't limit unit to be bytes, they don't limit verbs also; The point is that adding a custom verb like COUNT force requesters to know it, firewalls to accept it. On the other hand, Content-Range header usage with a 'resource' as unit is a flat win for requester, doesn't break current API and won't break anything in any system dealing with the answer.

I feel like it's enough for a first implementation (I may be very wrong, but as json+ld implementation already has those info, it's more like populating this header), but in future a nice improvement is Range: header can also be used instead of a page query parameter, to have a complete pagination handling in headers. It allows also to give some love to the Accept-Ranges: header and maybe the OPTION verb to document how to deal with pagination (ranges is the correct name in fact)!

Resources (no pun intended):

• https://stackoverflow.com/questions/5393558/how-should-i-implement-a-count-verb-in-my-restful-web-service
• https://www.devdoc.net/web/developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range.html
• https://datatracker.ietf.org/doc/html/rfc7233
• https://datatracker.ietf.org/doc/html/rfc9110
• https://hackernoon.com/range-headers-what-are-they-and-how-to-use-them
• Digest version of RFC 9110:
  • [Client] Request Range headers:
    • Range
    • If-Range
  • [Server] Response for Range:
    • Header Content-Range
    • Header Accept-Ranges
    • When returning a full list of resources (for example 3 books in database, so a GET request will get the fullset) --> code 200 OK
    • When returning a partial list of resources (GET request on an API where the number of items > number of items per age) --> Code 206 Partial Content
    • When being requested out of range (request's Range header, but also applie on a classic request with pagination for example /books?page=999 if I have only 20 books) --> Code 416 Range Not Satisfiable

[Nayte91]

In API-Platform/core:

• unit: InflectorInterface::pluralize(Metadata::getShortName())
• Where is the content-type negociated in first place?
• PaginatorInterface
• Where to change the status code to 206? RespondProcessor ?

[soyuka]

• unit: InflectorInterface::pluralize(Metadata::getShortName())

Not sure why you'd need this let's use only the operation's name or something else (maybe that we can discuss this later on)

• Where is the content-type negociated in first place?

https://github.com/api-platform/core/blob/c08758c93346b5819ab924ba1e7fe144c74f0007/src/State/Provider/ContentNegotiationProvider.php

• PaginatorInterface

Use a PartialPaginator for this?

• Where to change the status code to 206? RespondProcessor ?

yes but only if the user didn't specify its own status

[Nayte91]

Not sure why you'd need this let's use only the operation's name or something else (maybe that we can discuss this later on)

The name of the range unit is needed for the HTTP header 'Content-Range' on response, as defined here. For example, if I call the bookshop demo API with this url https://demo.api-platform.com/books?page=2 then the response should contain a Content-Range: books 31-60/201
So I don't know if there is a better way to get the resource name, put it in plural, then in lowercase, than the getShortName() method of Metadata class?

yes but only if the user didn't specify its own status

Status code seems a bit more complicated or subtle to use it accordingly with RFC 9110; 206 should be sent only if client asks for a range that can't be responded as it:

• if client ask for the whole books collection, and the server has a limit of 30 books per response, then a 206 should be fired,
• if client ask for a given page, then server should fire a 200,
• if client, in any given way, ask for books 31-65 (so 35 books), then the server will fire 30 books and a 206.

So implementing this is can be cut in subtasks:

• tweaking the server response (status code, Content-Range, Accept-Ranges), independently,
• tweaking the server's controller to catch new headers and take decision with other params & controls (for example response format can be for now either controlled with Accept header or in URL; That should be the same for pagination/range control) independently.

[soyuka]

👍 let me know if you need help