Description
DatasetClient.listItems() used as an async iterable returns wrong results with clean / skipEmpty / skipHidden.
The API applies offset/limit to the raw item stream and filters afterwards, so a page can scan up to limit rows while returning fewer (or zero) items. x-apify-pagination-count reports the scanned window; the iterator ignores it. Two bugs in _listPaginatedFromCallback:
- Stops early —
items.length > 0 (src/base/api_client.ts:118) ends iteration on a fully filtered page, even though later pages hold matching items.
- Yields duplicates —
currentOffset += currentPage.items.length (src/base/api_client.ts:128) advances by returned instead of scanned items, so the next page re-scans rows already yielded. The header is discarded at src/resource_clients/dataset.ts:400.
Cursor iterators (listKeys, listRequests, paginateRequests) are unaffected — the API never returns an empty page with a usable cursor.
Fixed in the Python client in apify/apify-client-python#964. A JS fix must read x-apify-pagination-count separately for bookkeeping, since count in PaginatedList is public and intentionally set to data.length.
API-side root cause: apify/apify-core#27324.
Description from #1032
paginateRequests() returns as soon as a page carries no items (src/utils.ts:329), before looking at nextCursor. With filter: 'pending' or 'locked' the API can filter out a whole page while the cursor still points at more data, so the iterator truncates silently. listKeys() has the same stop condition (src/resource_clients/key_value_store.ts:215). Termination should rely on the cursor alone.
#979 states cursor iterators are unaffected — that does not hold for a server-side filter.
Python: apify/apify-client-python#964.
✍️ Drafted by Claude Code
Description
DatasetClient.listItems()used as an async iterable returns wrong results withclean/skipEmpty/skipHidden.The API applies
offset/limitto the raw item stream and filters afterwards, so a page can scan up tolimitrows while returning fewer (or zero) items.x-apify-pagination-countreports the scanned window; the iterator ignores it. Two bugs in_listPaginatedFromCallback:items.length > 0(src/base/api_client.ts:118) ends iteration on a fully filtered page, even though later pages hold matching items.currentOffset += currentPage.items.length(src/base/api_client.ts:128) advances by returned instead of scanned items, so the next page re-scans rows already yielded. The header is discarded atsrc/resource_clients/dataset.ts:400.Cursor iterators (
listKeys,listRequests,paginateRequests) are unaffected — the API never returns an empty page with a usable cursor.Fixed in the Python client in apify/apify-client-python#964. A JS fix must read
x-apify-pagination-countseparately for bookkeeping, sincecountinPaginatedListis public and intentionally set todata.length.API-side root cause: apify/apify-core#27324.
Description from #1032
paginateRequests()returns as soon as a page carries no items (src/utils.ts:329), before looking atnextCursor. Withfilter: 'pending'or'locked'the API can filter out a whole page while the cursor still points at more data, so the iterator truncates silently.listKeys()has the same stop condition (src/resource_clients/key_value_store.ts:215). Termination should rely on the cursor alone.#979 states cursor iterators are unaffected — that does not hold for a server-side
filter.Python: apify/apify-client-python#964.
✍️ Drafted by Claude Code