Skip to content

Commit 3c9b08f

Browse files
authored
Add pagination and expanded data set, piece, provider, and payment queries (#913)
* refactor: reorganize SDK implementation * fix(react): paginate warm storage data sets and pieces * fix: use cursor pagination for service queries * docs: document cursor-based pagination * refactor(core): centralize look-ahead pagination handling
1 parent 61169d1 commit 3c9b08f

59 files changed

Lines changed: 1937 additions & 1612 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/src/content/docs/core-concepts/storage-providers.mdx

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,10 +84,13 @@ import { createPublicClient, http } from "viem"
8484
import { calibration } from "@filoz/synapse-core/chains"
8585
const publicClient = createPublicClient({ chain: calibration, transport: http() })
8686
// ---cut---
87+
import { paginate } from "@filoz/synapse-core"
8788
import { getApprovedProviderIds } from "@filoz/synapse-core/warm-storage"
8889
import { getEndorsedProviderIds } from "@filoz/synapse-core/endorsements"
8990

90-
const approved = await getApprovedProviderIds(publicClient)
91+
const approved = await Array.fromAsync(
92+
paginate(({ cursor }) => getApprovedProviderIds(publicClient, { cursor }))
93+
)
9194
const endorsed = await getEndorsedProviderIds(publicClient)
9295

9396
console.log("Approved provider IDs:", approved)

docs/src/content/docs/developer-guides/migration-guide.md

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,103 @@ If you are coming from an earlier version of any of the Synapse packages, you wi
99

1010
---
1111

12+
## Unreleased
13+
14+
### Action: Migrate paginated reads to cursors and pages
15+
16+
Paginated actions in `@filoz/synapse-core` now share a bounded cursor interface:
17+
18+
```ts
19+
type PaginationOptions = {
20+
cursor?: bigint
21+
limit?: bigint
22+
}
23+
24+
type Page<T> = {
25+
items: T[]
26+
nextCursor?: bigint
27+
}
28+
```
29+
30+
Replace contract-specific `offset`, `hasMore`, and array result handling with `cursor`, `items`, and `nextCursor`. Treat `nextCursor` as opaque and pass it back unchanged. Omitting `limit` uses a bounded default; `limit: 0n` is now rejected instead of meaning “fetch everything.”
31+
32+
```ts
33+
// before
34+
const dataSets = await getClientDataSets(client, {
35+
address,
36+
offset: 0n,
37+
limit: 0n,
38+
})
39+
40+
// after: read one page
41+
const page = await getClientDataSets(client, {
42+
address,
43+
limit: 100n,
44+
})
45+
console.log(page.items)
46+
47+
const nextPage = page.nextCursor === undefined
48+
? undefined
49+
: await getClientDataSets(client, {
50+
address,
51+
cursor: page.nextCursor,
52+
limit: 100n,
53+
})
54+
```
55+
56+
Use the generic `paginate()` generator to traverse every page or accumulate all items:
57+
58+
```ts
59+
import { paginate } from '@filoz/synapse-core'
60+
import { getClientDataSets } from '@filoz/synapse-core/warm-storage'
61+
62+
for await (const dataSet of paginate(({ cursor }) =>
63+
getClientDataSets(client, { address, cursor })
64+
)) {
65+
console.log(dataSet.dataSetId)
66+
}
67+
68+
const allDataSets = await Array.fromAsync(
69+
paginate(({ cursor }) => getClientDataSets(client, { address, cursor }))
70+
)
71+
```
72+
73+
This result change applies to paginated payment rails, FWSS client data sets and approved providers, PDP pieces and CID matches, and service-provider registry queries. Payment rail pages additionally include `total`.
74+
75+
The `WarmStorageService.getClientDataSets()` and `getClientDataSetIds()` methods in `@filoz/synapse-sdk` expose the same page interface. Higher-level SDK methods whose names promise all results, such as provider listing and rail listing methods, continue to return complete arrays and paginate internally.
76+
77+
### Action: Replace `getActivePieces` with `getActivePiecesByCursor`
78+
79+
The offset-based `getActivePieces` action was removed. Use piece-ID cursor pagination instead:
80+
81+
```ts
82+
// before
83+
const result = await getActivePieces(client, {
84+
dataSetId,
85+
offset: 0n,
86+
limit: 100n,
87+
})
88+
89+
// after
90+
const page = await getActivePiecesByCursor(client, {
91+
dataSetId,
92+
limit: 100n,
93+
})
94+
95+
// iterate
96+
for await (const piece of paginate(({ cursor }) =>
97+
getActivePiecesByCursor(client, { dataSetId, cursor, limit: 100n })
98+
)) {
99+
console.log(piece.id, piece.cid)
100+
}
101+
```
102+
103+
`findPieceIdsByCid`, `getPieces`, and `getPiecesWithMetadata` also return pages and accept `cursor` rather than `startPieceId` or `offset` at the action level.
104+
105+
Raw `*Call` helpers in `@filoz/synapse-core` remain ABI-oriented: provide their required contract-facing `offset` or `startPieceId` and `limit` fields explicitly when constructing multicalls.
106+
107+
---
108+
12109
## 1.0.0
13110

14111
### Action: Replace `terminateDataSet` with `terminateService`

docs/src/content/docs/developer-guides/synapse-core.mdx

Lines changed: 76 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,79 @@ const walletClient = createWalletClient({
6868

6969
The `@filoz/synapse-core/chains` subpath exports chain definitions with all contract addresses pre-configured for Filecoin Mainnet (`mainnet`) and Filecoin testnet (`calibration`) networks.
7070

71+
## Pagination
72+
73+
Collection actions in Synapse Core return one bounded page at a time. They share the same options and result shapes:
74+
75+
```ts
76+
type PaginationOptions = {
77+
cursor?: bigint
78+
limit?: bigint
79+
}
80+
81+
type Page<T> = {
82+
items: T[]
83+
nextCursor?: bigint
84+
}
85+
```
86+
87+
Some actions also return a `total` alongside the page. A cursor is an opaque continuation value: pass the returned `nextCursor` back as `cursor` without calculating or incrementing it yourself. An omitted `limit` uses the action's bounded default, while an explicit limit must be greater than `0n`.
88+
89+
To read one page, call the action directly:
90+
91+
```ts twoslash
92+
// @lib: esnext,dom
93+
import { createPublicClient, http } from "viem"
94+
import { calibration } from "@filoz/synapse-core/chains"
95+
const publicClient = createPublicClient({ chain: calibration, transport: http() })
96+
const address = "0x0000000000000000000000000000000000000000"
97+
// ---cut---
98+
import { getClientDataSets } from "@filoz/synapse-core/warm-storage"
99+
100+
const page = await getClientDataSets(publicClient, {
101+
address,
102+
limit: 25n,
103+
})
104+
105+
console.log(page.items)
106+
107+
if (page.nextCursor !== undefined) {
108+
const nextPage = await getClientDataSets(publicClient, {
109+
address,
110+
cursor: page.nextCursor,
111+
limit: 25n,
112+
})
113+
console.log(nextPage.items)
114+
}
115+
```
116+
117+
Use `paginate()` when you want to traverse every page. It follows `nextCursor`, yields individual items, and rejects a repeated or non-advancing cursor. A normal `break` exits the `for await` loop, closes the generator, and prevents any further page requests.
118+
119+
```ts twoslash
120+
// @lib: esnext,dom
121+
import { createPublicClient, http } from "viem"
122+
import { calibration } from "@filoz/synapse-core/chains"
123+
const publicClient = createPublicClient({ chain: calibration, transport: http() })
124+
const address = "0x0000000000000000000000000000000000000000"
125+
// ---cut---
126+
import { paginate } from "@filoz/synapse-core"
127+
import { getClientDataSetIds } from "@filoz/synapse-core/warm-storage"
128+
129+
for await (const dataSetId of paginate(({ cursor }) =>
130+
getClientDataSetIds(publicClient, { address, cursor })
131+
)) {
132+
console.log(dataSetId)
133+
}
134+
135+
const allDataSetIds = await Array.fromAsync(
136+
paginate(({ cursor }) =>
137+
getClientDataSetIds(publicClient, { address, cursor })
138+
)
139+
)
140+
```
141+
142+
The exported `*Call` helpers remain literal ABI adapters for multicalls. They use contract-facing fields such as `offset` or `startPieceId` and require an explicit `limit`; cursor translation and default limits are handled by the corresponding action.
143+
71144
## Payments
72145

73146
Query account balances, deposit funds, manage operator approvals, and settle payment rails on the Filecoin Pay contract.
@@ -128,9 +201,9 @@ const publicClient = createPublicClient({ chain: calibration, transport: http()
128201
// ---cut---
129202
import * as warmStorage from "@filoz/synapse-core/warm-storage"
130203

131-
// List approved providers
132-
const providers = await warmStorage.getApprovedProviderIds(publicClient)
133-
console.log(providers) // bigint[] — approved provider IDs
204+
// Read one bounded page of approved provider IDs
205+
const page = await warmStorage.getApprovedProviderIds(publicClient)
206+
console.log(page.items, page.nextCursor)
134207
```
135208

136209
---

examples/cli/src/commands/datasets-create.ts

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import * as p from '@clack/prompts'
2+
import { paginate } from '@filoz/synapse-core'
23
import * as sp from '@filoz/synapse-core/sp'
34
import {
45
getPDPProvider,
@@ -78,12 +79,14 @@ async function selectProvider(
7879
spinner.start(`Fetching providers...`)
7980

8081
try {
81-
const result = await getPDPProviders(client)
82+
const providers = await Array.fromAsync(
83+
paginate(({ cursor }) => getPDPProviders(client, { cursor }))
84+
)
8285
spinner.stop(`Fetching providers complete`)
8386

8487
const provider = await p.select({
8588
message: 'Pick a provider to create a data set.',
86-
options: result.providers.map((provider) => ({
89+
options: providers.map((provider) => ({
8790
value: provider,
8891
label: `#${provider.id} - ${provider.serviceProvider} ${provider.pdp.serviceURL}`,
8992
})),
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
import * as p from '@clack/prompts'
2+
import { getPdpDataSet } from '@filoz/synapse-core/warm-storage'
3+
import { type Command, command } from 'cleye'
4+
import { stringify } from 'viem'
5+
import { publicClient } from '../client.ts'
6+
import { globalFlags } from '../flags.ts'
7+
8+
export const datasetsInfo: Command = command(
9+
{
10+
name: 'datasets-info',
11+
description: 'Show data set information',
12+
alias: 'di',
13+
parameters: ['<dataSetId>'],
14+
flags: {
15+
...globalFlags,
16+
},
17+
help: {
18+
description: 'Show the Filecoin Warm Storage Service data for a data set',
19+
examples: [
20+
'synapse-cli datasets-info 123',
21+
'synapse-cli datasets-info 123 --chain 314',
22+
],
23+
},
24+
},
25+
async (argv) => {
26+
const client = publicClient(argv.flags.chain)
27+
28+
try {
29+
const dataSetId = BigInt(argv._.dataSetId)
30+
const dataSet = await getPdpDataSet(client, { dataSetId })
31+
if (dataSet == null) {
32+
throw new Error(`Data set #${dataSetId} not found`)
33+
}
34+
35+
p.log.message(stringify(dataSet, undefined, 2))
36+
} catch (error) {
37+
if (argv.flags.debug) {
38+
console.error(error)
39+
} else {
40+
p.log.error((error as Error).message)
41+
}
42+
}
43+
}
44+
)

examples/cli/src/commands/datasets.ts

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
import * as p from '@clack/prompts'
2+
import { paginate } from '@filoz/synapse-core'
23
import { getPdpDataSets } from '@filoz/synapse-core/warm-storage'
34
import { type Command, command } from 'cleye'
45
import { getBlockNumber } from 'viem/actions'
56
import { privateKeyClient } from '../client.ts'
6-
import { globalFlags } from '../flags.ts'
7+
import { Address, globalFlags } from '../flags.ts'
78

89
export const datasets: Command = command(
910
{
@@ -12,6 +13,11 @@ export const datasets: Command = command(
1213
alias: 'ds',
1314
flags: {
1415
...globalFlags,
16+
address: {
17+
type: Address,
18+
description: 'The address to list data sets for',
19+
default: undefined,
20+
},
1521
},
1622
help: {
1723
description: 'List all data sets',
@@ -21,29 +27,25 @@ export const datasets: Command = command(
2127
async (argv) => {
2228
const { client } = privateKeyClient(argv.flags.chain)
2329

24-
const spinner = p.spinner()
25-
2630
const blockNumber = await getBlockNumber(client)
31+
const address = argv.flags.address ?? client.account.address
2732

28-
spinner.start('Listing data sets...')
33+
p.log.info('Listing data sets...')
2934
try {
30-
const dataSets = await getPdpDataSets(client, {
31-
address: client.account.address,
32-
})
33-
spinner.stop('Data sets:')
34-
dataSets.forEach(async (dataSet) => {
35-
p.log.info(
36-
`#${dataSet.dataSetId} ${new URL(dataSet.provider.pdp.serviceURL).hostname} #${dataSet.providerId} ${dataSet.pdpEndEpoch > 0n ? `Terminating at epoch ${dataSet.pdpEndEpoch}` : ''}${dataSet.cdn ? ' CDN' : ''}`,
35+
for await (const item of paginate(({ cursor }) =>
36+
getPdpDataSets(client, { address, cursor })
37+
)) {
38+
p.log.step(
39+
`#${item.dataSetId} ${new URL(item.provider.pdp.serviceURL).hostname} #${item.providerId} ${item.pdpEndEpoch > 0n ? `Terminating at epoch ${item.pdpEndEpoch}` : ''}${item.cdn ? ' CDN' : ''}`,
3740
{ spacing: 0 }
3841
)
39-
})
42+
}
4043
p.log.warn(`Block number: ${blockNumber}`)
4144
} catch (error) {
4245
if (argv.flags.debug) {
43-
spinner.clear()
4446
console.error(error)
4547
} else {
46-
spinner.error((error as Error).message)
48+
p.log.error((error as Error).message)
4749
}
4850
}
4951
}

examples/cli/src/commands/get-sp-peer-ids.ts

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import * as p from '@clack/prompts'
2+
import { paginate } from '@filoz/synapse-core'
23
import { getPDPProviders } from '@filoz/synapse-core/sp-registry'
34
import { type Command, command } from 'cleye'
45
import { publicClient } from '../client.ts'
@@ -53,8 +54,10 @@ export const getSpPeerIds: Command = command(
5354
)
5455

5556
async function fetchProviderPeerIds(client: ReturnType<typeof publicClient>) {
56-
const result = await getPDPProviders(client)
57-
return result.providers.map((provider) => ({
57+
const providers = await Array.fromAsync(
58+
paginate(({ cursor }) => getPDPProviders(client, { cursor }))
59+
)
60+
return providers.map((provider) => ({
5861
providerId: provider.id,
5962
name: provider.name,
6063
ipniPeerId: provider.pdp.ipniPeerId,

0 commit comments

Comments
 (0)