Skip to content

Describe handling of paged resources, add helper functions #74

@rafalkrupinski

Description

@rafalkrupinski

Paging, even though has many different implementations, boils down to a couple of elements

  1. extracting paging parameters from response or generating them

    Paging parameters may be returned in header or headers, body or not returned at all.

    In case of parameters not being returned, the client keeps track of either page offset or page number and optionally page size.

  2. extracting parameters from links

    Some servers return links instead of parameters. Typically these links lead to the same endpoint and only add paging parameters.

    As long as server returns paging parameters in some form, their semantic (cursor, offset or page number) doesn't matter. They are simply passed from the previous response to the following request.
    Only when server doesn't return any parameters, the client is expected calculate offsets or page numbers and stop iteration when it receives response with no items.

  3. supplying the paging parameters to the next requests.

    The paging parameters may need to be supplied to query, path parameters or headers, but Lapidary treats in the same way, as function parameters.

Paging will be handled as function external to ApiClient:

def loop_pages(fn: Callable[..., [R]], param_extractor: Callable[[Any], dict])->Callable[[...],AsyncIterator[R]]:
  """Accept an operation method and an extractor function, and return AsyncGenerator that
  1. calls `fn` with provided parameters,
  5. yields the result and passes it to `param_extractor` to generate additional parameters for the next call
  6. call fn with additional params
  7. as long as param_extractor returns params, goto 3
  """

def extract_params(url: str, params: Sequence[str])->dict:
"""
extract query parameters from a URL
"""

async def explode_list(iter: AsyncIterator, list_extractor: Callable = lambda x:x)-> AsyncIterator:
  """If the operation method returns a list or an object that wraps a list, turn the async iterator returned by loop_pages to a single continuous iterator of items contained in those lists.
Optionally accept `prefetch: int` parameter
"""

Add helper function to extract query parameters from links

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions