[FEATURE] Pagination support for collections and records #22
Description
The current SDK implementation fetches all collections and all records at once, which causes performance issues when dealing with large datasets. Applications using the SDK (like our Collections Explorer) have to fetch all N collections created and/or all M records for a collection, which can fail if there are hundreds of thousands returned.
Methods
-
readCollections() - Returns ALL collections with no limit or pagination parameters
-
findData({ collection, filter }) - Returns ALL matching documents with no pagination
-
tailData(collection, limit) - Returns the last N documents
- tailData does accept a limit, but it only retrieves the most recent records - it doesn't support offset-based pagination needed for navigating through all records
Proposed Feature
Collection pagination
readCollections(options?: {
limit?: number;
offset?: number; // or cursor-based: after?: string
orderBy?: { field: string; direction: 'asc' | 'desc' };
}): Promise<{
data: Collection[];
pagination: {
total: number;
limit: number;
offset: number;
hasNext: boolean;
hasPrev: boolean;
}
}>
Record pagination
findData(params: {
collection: Uuid;
filter: Record<string, unknown>;
limit?: number;
offset?: number; // or cursor-based: after?: string
orderBy?: { field: string; direction: 'asc' | 'desc' };
}): Promise<{
data: Record<string, unknown>[];
pagination: {
total: number;
limit: number;
offset: number;
hasNext: boolean;
hasPrev: boolean;
}
}>
Use Case Example
As a dev building an app I want to be able to
- Load data incrementally as users navigate
- Display accurate "showing X of Y results" information
- Implement proper pagination controls
- Handle large datasets without performance degradation
Support the following:
const page1 = await client.readCollections({ limit: 20, offset: 0 });
console.log(`Showing ${page1.data.length} of ${page1.pagination.total} collections`);
// Fetch page 3 of data (items 100-149)
const records = await client.findData({
collection: collectionId,
filter: {},
limit: 50,
offset: 100,
orderBy: { field: '_id', direction: 'asc' }
});
// Check if there are more pages
if (records.pagination.hasNext) {
// Fetch next page...
}