Add pagination and expanded data set, piece, provider, and payment queries by hugomrdias · Pull Request #913 · FilOzone/synapse-sdk

hugomrdias · 2026-08-12T15:50:59Z

Summary

Add reusable pagination utilities and paginated PDP, warm-storage, provider registry, and payment queries
Return cursor-paginated dataset, piece, provider, and payment results from synapse-core, and update synapse-sdk consumers to traverse those pages explicitly
Update CLI commands and flags to support the new query capabilities
Add mocks and coverage for pagination and newly supported queries

Pagination examples

Read one bounded page and pass the returned cursor back unchanged:

const page = await getClientDataSets(client, {
  address,
  limit: 100n,
})

if (page.nextCursor !== undefined) {
  const nextPage = await getClientDataSets(client, {
    address,
    cursor: page.nextCursor,
    limit: 100n,
  })
}

Iterate over every item with the shared paginate() generator:

import { paginate } from "@filoz/synapse-core"

for await (const dataSet of paginate(({ cursor }) =>
  getClientDataSets(client, { address, cursor })
)) {
  console.log(dataSet.dataSetId)
}

Accumulate all items when a complete array is required:

const dataSets = await Array.fromAsync(
  paginate(({ cursor }) => getClientDataSets(client, { address, cursor }))
)

Cursors are opaque continuation values. Each action uses a bounded default when limit is omitted, and rejects limit: 0n.

Testing

Added unit tests covering pagination, data sets, pieces, providers, payment rails, and storage queries
Not run (not requested)

cloudflare-workers-and-pages · 2026-08-12T15:51:05Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	synapse-dev	`5b0d94c`	Commit Preview URL Branch Preview URL	Aug 18 2026, 02:59 PM

SgtPooki

in general lgtm.. huge change, but everything seems legit.. a few callouts that aren't necessarily blocking but could make some things better.

SgtPooki · 2026-08-13T14:39:42Z

+})
+
+// after
+const page = await getActivePiecesByCursor(client, {


the name here feels weird, when we're not passing a cursor. the rename seems unnecessary. can we leave it as getActivePieces ?

I’d prefer to keep this name. Synapse Core actions intentionally match contract function names, and the underlying endpoint is getActivePiecesByCursor. It also avoids conflating this endpoint with the older offset-based getActivePieces contract method.

SgtPooki · 2026-08-13T14:46:45Z

+
+`findPieceIdsByCid`, `getPieces`, and `getPiecesWithMetadata` also return pages and accept `cursor` rather than `startPieceId` or `offset` at the action level.
+
+Raw `*Call` helpers remain ABI-oriented: provide their required contract-facing `offset` or `startPieceId` and `limit` fields explicitly when constructing multicalls.


and these raw call methods are just in synapse-core, yes? I think it would be nice to normalize synapse-sdk being the "polished" surface and synapse-core being the lego bricks.

Yes, these raw ABI-oriented helpers are part of @filoz/synapse-core. The higher-level SDK remains the polished surface and either exposes normalized pages or fully traversed arrays depending on the method. I’ll make that package boundary explicit in the migration guide.

SgtPooki · 2026-08-13T15:11:04Z

+}
+```
+
+Use `paginate()` when you want to traverse every page. It follows `nextCursor`, yields individual items, and rejects a repeated or non-advancing cursor.


is it possible to break out of a paginate inner loop?

Yes. paginate() is an async generator, so a normal break exits the loop and closes the generator; it won’t fetch another page.

SgtPooki · 2026-08-13T16:16:41Z

+  const hasMore = BigInt(data.length) > limit
+  return {
+    items: Array.from(hasMore ? data.slice(0, -1) : data),
+    ...(hasMore ? { nextCursor: cursor + limit } : {}),
  }


this is duplicated in three places, and findPieceIdsByCid, getApprovedProviderIds, and getProvidersByProductType are missing limit+1 boundary testing, which means any individual implementation can drift.

Can we pull this out into a shared helper and then test that shared helper so we are sure each implementation is doing what it's supposed to?

SgtPooki · 2026-08-13T16:29:59Z

+  const [pdpProviders, approvedProviders] = await Promise.all([
+    Array.fromAsync(
+      paginate(({ cursor }) =>
+        getPDPProviders(client, { onlyActive: true, cursor, contractAddress: options.contractAddress })
+      )
+    ),
+    Array.fromAsync(paginate(({ cursor }) => getApprovedProviderIds(client, { cursor }))),
+  ])
+  return pdpProviders.filter((provider) => approvedProviders.includes(provider.id))


suggestion (non-blocking): getApprovedPDPProviders is only exercised indirectly, via fetch-provider-selection-input.test.ts, and only against single-page responses.. the basic preset never returns limit+1 items or hasMore: true, so paginate()'s multi-page path never runs against a real action anywhere in synapse-core (pagination.test.ts uses an in-memory stub).

The old get-client-data-sets.test.ts covered real multi-call traversal (150 items across 100+50 RPC calls) and was deleted with the old API, so multi-page coverage is net lower after this PR.

Can we add a direct test here with a mock returning 2+ pages? That covers getApprovedPDPProviders itself (including the approved-ids intersection) and restores the traversal coverage in one shot.

SgtPooki · 2026-08-13T16:32:34Z

              ],
            },
          ],
          hasMore: false,


this getProvidersByProductType mock ignores offset/limit and hardcodes hasMore:false.. we could extend this so we can write tests to help catch any cursor handling regression

rvagg · 2026-08-18T10:17:40Z

+  limit: 100n,
+})
+
+for await (const piece of paginate(({ cursor }) =>


Suggested change

for await (const piece of paginate(({ cursor }) =>

// iterate

for await (const piece of paginate(({ cursor }) =>

currently reads like it's part of // after

rvagg · 2026-08-18T10:27:07Z

-import { readPdpDataSetInfo } from './get-pdp-data-set.ts'
-import type { PdpDataSet } from './types.ts'
+const ENRICHMENT_BATCH_SIZE = 20
+const DATA_SET_CALL_COUNT = 4


I think this one deserves a comment, it's going to be very easy to get this out of syncwith the number of calls

rvagg

well I don't hate it, I think that's what you wanted to hear?

needs a ! in a commit somewhere to bump major I guess

also utils/sp-tool.js will be broken after this, may as well fix it now, also core-concepts/storage-providers.mdx could do with a look, it has getApprovedProviderIds calls

github-project-automation Bot added this to FOC Aug 12, 2026

github-project-automation Bot moved this to 📌 Triage in FOC Aug 12, 2026

hugomrdias marked this pull request as ready for review August 12, 2026 17:22

hugomrdias requested review from a team, BigLep, jennijuju, rjan90 and rvagg as code owners August 12, 2026 17:22

hugomrdias self-assigned this Aug 12, 2026

This was linked to issues Aug 12, 2026

getPdpDataSet fails on large data sets because getActivePieceCount reverts inside multicall #851

Closed

Normalize pagination interface #753

Closed

SgtPooki approved these changes Aug 13, 2026

View reviewed changes

github-project-automation Bot moved this from 📌 Triage to ✔️ Approved by reviewer in FOC Aug 13, 2026

rvagg reviewed Aug 18, 2026

View reviewed changes

rvagg approved these changes Aug 18, 2026

View reviewed changes

hugomrdias removed request for a team, BigLep, jennijuju and rjan90 August 18, 2026 11:38

hugomrdias added 5 commits August 18, 2026 15:45

refactor: reorganize SDK implementation

74f32b7

fix(react): paginate warm storage data sets and pieces

e888dfa

fix: use cursor pagination for service queries

9259aa2

docs: document cursor-based pagination

f8b5d6e

refactor(core): centralize look-ahead pagination handling

5b0d94c

hugomrdias force-pushed the hugomrdias/753 branch from 2a7f325 to 5b0d94c Compare August 18, 2026 14:45

hugomrdias merged commit 3c9b08f into master Aug 20, 2026
13 checks passed

hugomrdias deleted the hugomrdias/753 branch August 20, 2026 10:37

github-project-automation Bot moved this from ✔️ Approved by reviewer to 🎉 Done in FOC Aug 20, 2026

hugomrdias added a commit that referenced this pull request Aug 20, 2026

feat!: actions pagination and active piece count removal (#914, #913)

b1fdbe9

FilOzzy mentioned this pull request Aug 20, 2026

chore(master): release synapse-core 0.8.0 #908

Merged

hugomrdias mentioned this pull request Aug 21, 2026

Replace offset-based active piece pagination with getActivePiecesByCursor #848

Closed

5 tasks


		`findPieceIdsByCid`, `getPieces`, and `getPiecesWithMetadata` also return pages and accept `cursor` rather than `startPieceId` or `offset` at the action level.

		Raw `*Call` helpers remain ABI-oriented: provide their required contract-facing `offset` or `startPieceId` and `limit` fields explicitly when constructing multicalls.

	for await (const piece of paginate(({ cursor }) =>
	// iterate
	for await (const piece of paginate(({ cursor }) =>

Uh oh!

Conversation

hugomrdias commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Pagination examples

Testing

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SgtPooki Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SgtPooki Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rvagg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hugomrdias commented Aug 12, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Aug 12, 2026 •

edited

Loading

SgtPooki Aug 13, 2026 •

edited

Loading

SgtPooki Aug 13, 2026 •

edited

Loading