Handling paginated APIs with Mutiny

[giscus[bot]]

Handling paginated APIs with Mutiny

Quarkus: Supersonic Subatomic Java

https://quarkus.io/blog/mutiny-pagination/

[cobar79]

Took a couple of days to find this, but exactly what I needed for https://ucdpapi.pcr.uu.se/api/gedevents/22.1?pagesize=1000

[cobar79]

How would you convert this to concurrently pull a configurable amount of pages?
In my case https://ucdpapi.pcr.uu.se/api/gedevents/22.1?pagesize=1000 is 262 pages and 261K worth of details.

[tomschulze]

Thanks for the write-up! This helped me a lot for retrieving results from a GraphQL API that uses cursor-based pagination. Tmarven's stackoverflow answer helped me retrieving the cursor from the response.

[IvanPuntev]

Here is an example how to handle different pagination #36681

[Marcuse7]

The hosted Punk API will shut down May 1^st, 2024. You could run it locally, though.

[Darkvater]

Well, the example is handy, but if you actually dig deeper you see it has a performance penalty. To retrieve all pages this implementation always does 1 extra call as it only knows when to stop when the size of beers are finished.

This could be fine in some scenarios, but in others where you know the page size you can optimise it as you can already stop when returned beers < page size as you know that is the last page. To do this one should change the stop condition from .until(List::isEmpty) to .whilst( it -> it.size >= pageSize). However in the above scenario for some reason Mutiny has already scheduled the call to the second page even before the first page was retrieved (and then dropping it). The outcome is the same, but we did do one extra call still; at least on smallrye.reactive.mutiny 2.9.4.

Would be great to see an example which works properly knowing the page size in advance and doign the optimal number of calls.