Native Pagination Support for File Listing in OpenDAL

[cbisht31]

In order to iterate over large number of files in a directory, it becomes challenging to handle large datasets efficiently. Does openDAL supports pagination or similar mechanism to make this efficient ?

[Xuanwo]

Hi, thank you for using OpenDAL.

OpenDAL uses pagination internally; for example, with S3, it lists 1,000 entries per request.

Would you like to share more details about this? Which service are you using? What pagination you expect?

[cbisht31]

Thank you for the clarification regarding OpenDAL's internal pagination.

I'm using S3 and wanted to know how can I add offset and limit functionality to this code. The API should support pagination for the frontend. For example, I want to skip the first offset files and return only limit files in the response. Does OpenDAL provide a way to achieve this directly?
Also, is there a way to get the total count of files in a directory without listing all of them? For example, does OpenDAL provide a count function or a similar method to retrieve the total number of files in a directory? Here is the code snippet where I wish to add:

async def list_files_with_metainfo(operator: opendal.Operator, subpath: str) -> List[dict]:
    """
    List files at the given subpath and retrieve full metainfo for each file.
    """
    metainfo_list = []
    try:
        entries = await operator.list(subpath)
        async for entry in entries:
            name = entry.name if hasattr(entry, 'name') else str(entry)
            full_path = entry.path
            is_dir = False
            error = None
            try:
                stat_info = await operator.stat(full_path)
                if hasattr(stat_info, 'mode'):
                    is_dir = stat_info.mode.is_dir()
            except Exception as e:
                error = str(e)
            file_info = {
                "name": name,
                "path": full_path,
                "is_dir": is_dir,
                "error": error
            }
            metainfo_list.append(file_info)
    except Exception as e:
        print(f"Error: {e}")
    return metainfo_list

[Xuanwo]

I'm using S3 and wanted to know how can I add offset and limit functionality to this code. The API should support pagination for the frontend.

opendal rust core provides an option called start-after, users can start list from offset in this way:

let lister = op.lister_with(path).start_after(offset).await?;

But this feature seems not exported to python yet. Let's create an issue to track it.

Also, is there a way to get the total count of files in a directory without listing all of them?

Sadly, object storage services like s3 doesn't provide feature like this. We will need to list them all to get the count.

[cbisht31]

Thank you! I'll create an issue for the same.

[cbisht31]

Hi @Xuanwo,

Hope you doing well, just wanted check whether start-after feature is there on python yet?

Thanks!