-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Paging, even though has many different implementations, boils down to a couple of elements
-
extracting paging parameters from response or generating them
Paging parameters may be returned in header or headers, body or not returned at all.
In case of parameters not being returned, the client keeps track of either page offset or page number and optionally page size.
-
extracting parameters from links
Some servers return links instead of parameters. Typically these links lead to the same endpoint and only add paging parameters.
As long as server returns paging parameters in some form, their semantic (cursor, offset or page number) doesn't matter. They are simply passed from the previous response to the following request.
Only when server doesn't return any parameters, the client is expected calculate offsets or page numbers and stop iteration when it receives response with no items. -
supplying the paging parameters to the next requests.
The paging parameters may need to be supplied to query, path parameters or headers, but Lapidary treats in the same way, as function parameters.
Paging will be handled as function external to ApiClient:
def loop_pages(fn: Callable[..., [R]], param_extractor: Callable[[Any], dict])->Callable[[...],AsyncIterator[R]]:
"""Accept an operation method and an extractor function, and return AsyncGenerator that
1. calls `fn` with provided parameters,
5. yields the result and passes it to `param_extractor` to generate additional parameters for the next call
6. call fn with additional params
7. as long as param_extractor returns params, goto 3
"""
def extract_params(url: str, params: Sequence[str])->dict:
"""
extract query parameters from a URL
"""
async def explode_list(iter: AsyncIterator, list_extractor: Callable = lambda x:x)-> AsyncIterator:
"""If the operation method returns a list or an object that wraps a list, turn the async iterator returned by loop_pages to a single continuous iterator of items contained in those lists.
Optionally accept `prefetch: int` parameter
"""Add helper function to extract query parameters from links