Skip to content

Commit c71be31

Browse files
committed
fix(synapse-sdk): address batching review feedback
1 parent d244dc2 commit c71be31

4 files changed

Lines changed: 87 additions & 4 deletions

File tree

docs/src/content/docs/developer-guides/storage/upload-pipeline.mdx

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,43 @@ The result contains:
5555
- **`copies`** - array of successful copies, each with `providerId`, `dataSetId`, `pieceId`, `role` (`'primary'` or `'secondary'`), `retrievalUrl`, and `isNewDataSet`
5656
- **`failedAttempts`** - providers that were tried but did not produce a copy. The SDK retries failed secondaries with alternate providers, so a non-empty array often just means a provider was swapped out. These are diagnostic, check `complete` for the actual outcome.
5757

58+
### Uploading Multiple Files
59+
60+
Piece batching is enabled by default. Compatible uploads that run concurrently can share an on-chain transaction when they use the same provider and data set. Each provider maintains its own batch.
61+
62+
Start the uploads together to give them an opportunity to join the same batch:
63+
64+
```ts
65+
const results = await Promise.all(
66+
files.map((file) => synapse.storage.upload(file))
67+
)
68+
```
69+
70+
Sequential uploads cannot share a batch because each call waits for its own on-chain confirmation before the next call starts:
71+
72+
```ts
73+
for (const file of files) {
74+
await synapse.storage.upload(file)
75+
}
76+
```
77+
78+
By default, the SDK submits a batch after a zero-delay window. You can instead hold compatible pieces until the transaction size limit is reached or you explicitly flush them:
79+
80+
```ts
81+
const synapse = Synapse.create({
82+
account: privateKeyToAccount("0x..."),
83+
pieceBatching: { wait: { kind: "limiter" } },
84+
})
85+
86+
const uploads = files.map((file) => synapse.storage.upload(file))
87+
await synapse.storage.flush()
88+
const results = await Promise.all(uploads)
89+
```
90+
91+
`flush()` waits for accepted uploads and pulls to finish parking, then submits their pending batch windows. It does not report whether every upload was submitted or confirmed successfully; always await the individual upload promises for results and errors.
92+
93+
Set `pieceBatching: false` when creating `Synapse` to disable batching. Use the [split operations](#split-operations) when you need manual control over provider selection, signing, or each store, pull, and commit phase.
94+
5895
### Upload with Metadata
5996

6097
Attach metadata to organize uploads. The SDK reuses existing data sets when metadata matches, avoiding duplicate payment rails:

packages/synapse-sdk/src/storage/manager.ts

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -198,9 +198,14 @@ export class StorageManager {
198198
* to determine overall success. Don't use `failedAttempts.length` as a failure
199199
* signal as `failedAttempts` exists as a diagnostic for intermediate failures.
200200
*
201+
* Batching is enabled by default. Compatible concurrent calls can share
202+
* on-chain transactions, while sequentially awaiting each call prevents those
203+
* uploads from joining the same batch.
204+
*
201205
* For large files, prefer streaming to minimize memory usage.
202206
*
203-
* For uploading multiple files, use the split operations API directly:
207+
* For manual control over providers, signing, or individual phases, use the
208+
* split operations API directly:
204209
* createContexts() -> store() -> presignForCommit() -> pull() -> commit()
205210
*
206211
* @param data - Raw bytes (Uint8Array) or ReadableStream to upload
@@ -392,7 +397,15 @@ export class StorageManager {
392397
}
393398
}
394399

395-
/** Flush all piece windows currently accepted by this Synapse instance. */
400+
/**
401+
* Submit all piece-batch windows currently accepted by this Synapse instance.
402+
*
403+
* Waits for in-progress uploads and pulls to finish parking before submitting
404+
* their pending windows. Resolving does not mean every upload was submitted or
405+
* confirmed successfully; failures are reported by the individual upload
406+
* promises, which callers must also await. This is a no-op when batching is
407+
* disabled.
408+
*/
396409
async flush(): Promise<void> {
397410
await getPieceBatchingService(this._synapse)?.flush()
398411
}

packages/synapse-sdk/src/storage/piece-batching.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -162,15 +162,15 @@ export class PieceBatchingService {
162162
const key = this.key(context)
163163
const existing = this.entries.get(key)
164164
if (existing != null) {
165-
void existing.then((entry) => {
165+
return existing.then((entry) => {
166166
const dataSet = entry.batcher.dataSet
167167
if (dataSet == null) {
168168
entry.contexts.add(context)
169169
} else {
170170
context.syncBatcherDataSet(dataSet.dataSetId, dataSet.clientDataSetId)
171171
}
172+
return entry
172173
})
173-
return existing
174174
}
175175

176176
const created = this.createEntry(context)

packages/synapse-sdk/src/test/storage-upload.test.ts

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ import { setup } from 'iso-web/msw'
1212
import { HttpResponse, http } from 'msw'
1313
import { createWalletClient, http as viemHttp } from 'viem'
1414
import { privateKeyToAccount } from 'viem/accounts'
15+
import type { StorageContext } from '../storage/context.ts'
16+
import { PieceBatchingService } from '../storage/piece-batching.ts'
1517
import { Synapse } from '../synapse.ts'
1618
import type { PieceCID } from '../types.ts'
1719
import { SIZE_CONSTANTS } from '../utils/constants.ts'
@@ -180,6 +182,37 @@ describe('Storage Upload', () => {
180182
)
181183
})
182184

185+
it('should reject concurrent uploads cleanly when shared batch entry creation fails', async () => {
186+
const expected = new Error('entry creation failed')
187+
let rejectEntry: (error: Error) => void = () => undefined
188+
const entryFailure = new Promise<never>((_resolve, reject) => {
189+
rejectEntry = reject
190+
})
191+
const context = {
192+
dataSetId: 1n,
193+
provider: { id: 1n },
194+
getBatcherDataSet: () => entryFailure,
195+
} as unknown as StorageContext
196+
const batching = new PieceBatchingService(new Synapse({ client, source: null }), {})
197+
198+
const uploads = [
199+
batching.upload(context, { data: new Uint8Array(127) }).committed,
200+
batching.upload(context, { data: new Uint8Array(128) }).committed,
201+
]
202+
rejectEntry(expected)
203+
204+
const results = await Promise.allSettled(uploads)
205+
assert.deepEqual(
206+
results.map((result) => result.status),
207+
['rejected', 'rejected']
208+
)
209+
for (const result of results) {
210+
if (result.status === 'rejected') {
211+
assert.strictEqual(result.reason, expected)
212+
}
213+
}
214+
})
215+
183216
it('should expose flush for limiter-based batching', async () => {
184217
let addPiecesCalls = 0
185218
const txHash = '0xabcdef1234567890abcdef1234567890abcdef1234567890abcdef123456'

0 commit comments

Comments
 (0)