feat: multi sync example#1622
Conversation
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
…sync flag from run-15 usage Multi-sync is now the default for start:localnet; --no-multi-sync starts single-synchronizer debug mode. Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
…out test token Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
| } | ||
| ) | ||
|
|
||
| const signature = signTransactionHash( |
There was a problem hiding this comment.
The reason why we've split up the prepare/sign/execute functions is to enable offline signing. Could we move the synchronizerIds as part of the CreatePartyOptions so that on the sdk.party.external.create call this generates all of the topology transactions to be signed and then the SignedpartyCreationService takes in an array of ExecuteOptions?
There was a problem hiding this comment.
Or disregard this if offline signing is supposed to be enabled in a separate phase or not supported at all as per this
There was a problem hiding this comment.
I see the problem - but I think offline signing with multiple synchronizers is a bigger change -> separate PR/ design.
| * `KNOWN_PACKAGE_VERSION`. Since the already-vetted package is resolved by | ||
| * package-name at command-submission time, it is safe to reuse it and continue. | ||
| */ | ||
| async function vetPackageIdempotent( |
There was a problem hiding this comment.
Could this be added to the dar namespace in the sdk?
There was a problem hiding this comment.
logic moved to existing funtion
| * @param darBytes - Raw DAR file bytes. | ||
| * @param synchronizerId - The synchronizer on which the package should be vetted. | ||
| */ | ||
| export async function vetDar( |
There was a problem hiding this comment.
Is there a reason why this isn't part of the dar namespace where it can be called with sdk.ledger.dar.vet and then we don't have to pass the provider in as an arg (it will just reuse the provider initialized in the sdk) ?
There was a problem hiding this comment.
moved
also discovered that There is DarService which is duplicate of DarNamespace (comment added)
| const events = unassignResponse.reassignment?.events ?? [] | ||
| const unassignedEvent = events.find((e) => 'JsUnassignedEvent' in e) | ||
| if (!unassignedEvent || !('JsUnassignedEvent' in unassignedEvent)) { | ||
| throw new Error( |
There was a problem hiding this comment.
nitpick: can we use this.ctx.error.throw here instead
There was a problem hiding this comment.
done (+ in few other places)
| * @param synchronizerIds - Synchronizers to submit to in parallel | ||
| * @param privateKey - Key used to sign each prepared transaction | ||
| */ | ||
| public async executeOnSynchronizers( |
There was a problem hiding this comment.
I like this convenience method, but I do want the name of this to indicate that it's preparing (for all synchronizers), signing and executing. Do you think there could be any value in also having a bulk prepare method (prepareForAllSynchronizers) for the case of offline signing?
There was a problem hiding this comment.
method was renamed
as for prepareForAllSynchronizers I would add this in other PR
or just change this one to use few such methods
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
3435416 to
2b41e6b
Compare
Signed-off-by: jarekr-da <jaroslaw.ratajski@digitalasset.com>
93653ae to
1bee65e
Compare
meiersi-da
left a comment
There was a problem hiding this comment.
@jarekr-da @Viktor-Kalashnykov-da (FYI: @mziolekda @mjuchli-da ): good work on this! I have not completed the review yet, but the comments added so far are valuable on their own, and I'm out of time for today. I'm sharing them with you so you can start processing them.
| // Copyright (c) 2025-2026 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| export const AMULET_TEMPLATE_ID = '#splice-amulet:Splice.Amulet:Amulet' |
There was a problem hiding this comment.
I wonder what we need this for.
| @@ -0,0 +1,99 @@ | |||
| // Copyright (c) 2025-2026 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved. | |||
| // SPDX-License-Identifier: Apache-2.0 | |||
|
|
|||
There was a problem hiding this comment.
s/test-token/test-token-v1/ in the path, so there's space for test-token-v2, which will become useful later when implementing TSv2 support in the wallet
Also: the base path /core feels off. I'd consider this test-token test infrastructure, so I'd expect some there to be a /test/ directory somewhere in the path.
| } | ||
| } | ||
|
|
||
| /** Build an ExerciseCommand for TransferFactory_Transfer on a TokenRules contract. */ |
There was a problem hiding this comment.
This is off. You should use the wallet's default function to build a transfer call against token, which in turn uses the registry URL to fetch the necessary choice context.
What I'd expect to see is a HTTP server along the lines of https://github.com/canton-network/splice/blob/main/apps/scan/src/main/scala/org/lfdecentralizedtrust/splice/scan/admin/http/HttpTokenStandardTransferInstructionHandler.scala#L44 that serves the right choice context for test-token-v1; i.e., queries the TokenRules contract and serves it as the factoryId in the factoryId here: https://github.com/canton-network/splice/blob/e97400b86dd7c3ab0131752e774daabf2c0b5e4c/token-standard/splice-api-token-transfer-instruction-v1/openapi/transfer-instruction-v1.yaml#L193
| } | ||
|
|
||
| /** Build an ExerciseCommand that accepts a pending TransferInstruction (TokenTransferOffer). */ | ||
| export function buildAcceptTransferInstructionCommand(offerCid: string) { |
There was a problem hiding this comment.
Same here: should be done via the https://github.com/canton-network/wallet/blob/main/core/token-standard-service/src/token-standard-service.ts#L989, or a close neighbor of it -- I'm not sure what exactly is the top-level function that's exported.
| @@ -0,0 +1,306 @@ | |||
| -- Copyright (c) 2024 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved. | |||
There was a problem hiding this comment.
I'd strongly suggest to vendor in the .dar from splice, so you get the tested version, and don't create another copy that goes stale -- this one slightly stale, as I did recently fix some bugs.
I understand that the .dar is not yet officially released, but that shouldn't keep us from vendoring in the .dar itself. The official release will happen on July 2nd.
| // The settled holding lands on the global synchronizer; move it to the | ||
| // app-synchronizer before self-transferring there (mirrors Bob's flow). | ||
| if (aliceToken.synchronizerId !== appSynchronizerId) { | ||
| await appUserSdk.ledger.internal.reassign({ |
There was a problem hiding this comment.
as said above: remove all reassign commands.
| const OTC_TRADE_PROPOSAL_TEMPLATE_ID = | ||
| '#splice-token-test-trading-app:Splice.Testing.Apps.TradingApp:OTCTradeProposal' | ||
| const OTC_TRADE_TEMPLATE_ID = | ||
| '#splice-token-test-trading-app:Splice.Testing.Apps.TradingApp:OTCTrade' |
There was a problem hiding this comment.
consider moving this file to the test support code for a /core/test-trading-app-v1 to shrink the multi-sync example itself.
| if (match) return match.contractId | ||
| if (Date.now() >= deadline) { | ||
| throw new Error( | ||
| `OTCTradeProposal not visible to ${party} within ${PROPOSAL_POLL_TIMEOUT_MS}ms` |
There was a problem hiding this comment.
Do we really need to do this this way? Is there no better option in the .ts world to wait for a promise to be resolved within a certain amount of time?
| testTokenAllocationDisclosed: DisclosedContract | ||
| } | ||
|
|
||
| /** Withdraws both allocations in parallel after a settlement failure, returning funds to each party. */ |
There was a problem hiding this comment.
what do we need this for?
| } catch (settleError) { | ||
| logger.error( | ||
| { err: settleError }, | ||
| 'Settlement failed — withdrawing allocations to return funds' |
There was a problem hiding this comment.
I would have rather expected this is retried here; and for test code we'd expect it to never fail, wouldn't we?
meiersi-da
left a comment
There was a problem hiding this comment.
Many thanks for creating this example @jarekr-da @Viktor-Kalashnykov-da! (FYI: @mziolekda @tudor-da )
Discussion on automatic synchronizer routing
This example highlights important problems to consider wrt automatic synchronizer routing:
- optimal routing depends on follow-up transactions: consider the situation where Alice has one large holding contract assigned each to the GSync and the PSync. Alice now needs to create a small allocation to settle a trade. When minimizing reassignments, which synchronizer to choose depends on where the allocation is settled. That information is however not available to Canton's synchronizer router, which only sees the transaction that creates the allocation.
- client-side choices matter for optimal routing: in the above example, picking the holding as an input contract already determines the synchronizer to which to route the allocation creation so that reassignments are minimize: the one that the holding is assigned to. So the client constructing the command to create the allocation needs to pick the holding on the synchronizer on which the settlement ultimately happens to minimize the number of reassignments.
- reference data choices matter for optimal routing: creating an allocation requires reference data contracts (at least the allocation factory, often also credentials and instrument config contracts). They can usually not be reassigned by the EPN, and thus they fully determine the synchronizer of the transaction. So unless the token registry only offers the creation of allocations on a single synchronizer, the reference data contracts must be chosen such that they reside on the desired transaction synchronizer. Canton's synchronizer router in its current form cannot make this choice, as it cannot change the command that is interpreted.
@daravep: AFAIS, these three problems are all pointing in the same direction: applications do need to make explicit choices wrt what synchronizer to use for what step of their workflows. In particular with the current state of Canton where reference data can only exist on one synchronizer. You've pushed back in the past on having applications make this choice. Where do you stand these days?
The guideline that I'd recommend for building multi-sync applications in H2 2026 is:
- choose the target synchronizer for each workflow step as part of the apps' workflow design
- communicate this target synchronizer to the off-ledger registry APIs, so they can use that to deliver the right reference data for executing the action on that synchronizer
- submit the Ledger API command with a prescribed synchronizer, and let automatic reassignment take care of creating the reassignment transactions
- leave contracts by default assigned to the synchronizer where they were created
The third point likely also helps making traffic spend for reassignments more predictable.
It would be great if we could align on an initial guideline that we want to go live with, so the team can validate the guideline in this example.
Next steps
I think the example is already in a good shape. It has all the pieces that we need. We now just need to rejig them a bit to ensure it tests exactly what we want to test, and provides a solid foundation for future related tests.
Important changes to apply to the test setup:
- host every party on its own validator node to ensure co-hosting is not hiding a problem with the reassignment permissions
- adjust the example to follow the guidelines above (once we're aligned with @daravep)
I also suggest that you start chopping off standalone pieces that are required to build this example, and merge them on their own while we align on the guidelines for building multi-sync apps. For example,
- merge
core-test-token-v1on its own with a basic test that it supports creating and settling an allocation, and executing a transfer using the token standard support functions of the wallet SDK - merge the "multi-sync improvements" as a standalone refactoring of the SDK APIs
Follow-up: the TestTokenV1 is a greatly simplified token. I'd love to also run the example with TestTokenV2, which is a realistic implementation which uses separate holding contracts. I suspect that what we'll discover there is that it's important to make the settlement venue an observer of the locked holding; and that we really need Canton's automatic reassignments to take care of reassignments, as otherwise the app needs to know which contracts are referenced by a transaction to execute all reassignments ahead of command submission.
| // Disclose Bob's TestToken allocation to the TradingApp (sv participant): the | ||
| // allocation is created on the app-provider participant, so it may not yet be in the sv's ACS | ||
| // when settlement runs. Disclosing it makes settlement independent of cross- | ||
| // participant propagation timing. |
There was a problem hiding this comment.
That seems off: what we want to simulate is the trading app backend discovering the allocations on their participant node and then submit the command. I'd suggest to invest a bit into creating the utility function to make it easy to wait for the contracts to be visible and then use it to deal with the eventually consistent nature of the system under test.
Canton and splice use the eventually combinator for that: https://github.com/canton-network/splice/blob/25d885477f19ea5d436da69248991eaf1c04e0bf/canton/community/testing/src/main/scala/com/digitalasset/canton/BaseTest.scala#L461-L470
| { err: e }, | ||
| 'Settlement failed — compensation applied, funds returned' | ||
| ) | ||
| await bobSelfTransferToApp(setup, logger) |
There was a problem hiding this comment.
Not sure about that. What I'd recommend is to leave the Holding contracts free-floating on whatever synchronizer they end up. This minimizes traffic cost. What will though then be required is for the holding input selection to become synchronizer aware: preferably holdings on the target synchronizer should be selected to minimize reassignments.
Meta comment: this is evidence that relying purely on automatic synchronizer routing might not work, as doing this selection requires knowing the target synchronizer.
| { sdk: appProviderSdk, parties: [tokenAdmin.partyId] }, | ||
| { sdk: svSdk, parties: [tradingApp.partyId] }, | ||
| ]) | ||
| // ── Step 11: Self-transfer TestTokens back to app-synchronizer ───────────────── |
There was a problem hiding this comment.
I'd suggest to replace this with transfer from 'Alice' to a third party 'Charlie' that should be executed on the app synchronizer. This would test that doing transfers with the received tokens works; and there is no requirement to actively reassign the holdings.
|
|
||
| # Example details | ||
|
|
||
| The goal here is to show an exchange operation with a custom token (`TestToken`) that is deployed to a private / local `app-synchronizer`. |
There was a problem hiding this comment.
| The goal here is to show an exchange operation with a custom token (`TestToken`) that is deployed to a private / local `app-synchronizer`. | |
| The goal here is to show an exchange operation with a custom token (`TestToken`) that supports running workflows on both a private synchronizer and the global synchronizer. |
| The parties in the example are: | ||
|
|
||
| - **Alice** — app-user, hosted on the **app-user** participant. Holds Amulet, buys `TestToken`. | ||
| - **Bob** — app-provider, hosted on the **app-provider** participant. Holds `TestToken`, buys Amulet. |
There was a problem hiding this comment.
Bob is no app provider!
| /** | ||
| * Reassigns a contract from one synchronizer to another. | ||
| * Performs the two-phase Canton reassignment (Unassign → Assign) via | ||
| * `/v2/commands/submit-and-wait-for-reassignment`. |
There was a problem hiding this comment.
Is that available to external parties? We should understand how and when exactly it works. I suspect the test script currently exploits that particular parties are cohosted.
| const { submitter, contractId, source, target, skipIfAlreadyOn } = | ||
| params | ||
|
|
||
| if (skipIfAlreadyOn && source === target) { |
There was a problem hiding this comment.
When would you not want to skip the reassignment if the contract is already on the target? Do we even need the param?
| } | ||
| throw e | ||
| } | ||
|
|
There was a problem hiding this comment.
If the client crashes here, then there will be a contract left in an unassigned state. That is probably just part of the game. However it also means that this code here should be written such that it skips first completes any pending assignment; and then checks whether an assignment to the actual target synchronizer is still required.
Furthermore, this code is replicating code that already exists in Canton as part of the automatic synchronizer router. We should clarify why we can't use the Canton code.
| /** | ||
| * Prepares, signs, and executes the same command set on multiple synchronizers in parallel. | ||
| * Equivalent to calling `prepare(...).sign(privateKey).execute({ partyId })` for each | ||
| * synchronizer, but without repeating the command payload. | ||
| * @param options - Command options without a synchronizerId (it is provided per-element) | ||
| * @param synchronizerIds - Synchronizers to submit to in parallel | ||
| * @param privateKey - Key used to sign each prepared transaction | ||
| */ | ||
| public async prepareAndExecuteOnSynchronizers( | ||
| options: Omit<PrepareOptions, 'synchronizerId'>, | ||
| synchronizerIds: string[], | ||
| privateKey: PrivateKey | ||
| ): Promise<void> { | ||
| await Promise.all( | ||
| synchronizerIds.map((synchronizerId) => | ||
| this.prepare({ ...options, synchronizerId }) | ||
| .sign(privateKey) | ||
| .execute({ partyId: options.partyId }) | ||
| ) | ||
| ) |
There was a problem hiding this comment.
Do we really need this?
Multi-Synchronizer DvP Example - On ledger API - Part 1
Showcases automatic reassignment of Token from private synchronizer to global one, used on a realistic trade scenario with use of a custom Token.
New test:
https://github.com/canton-network/wallet/pull/1622/changes#diff-37db720ea457076fe6d04f6938d19c44e5dde3d7e1a5815953ff6b83199a1052
Design principles:
!!! We remove handling of synchronizers inside wallet-sdk.
Client code should be responsible of selecting
synchronizerIdin multi-sync scenarios.If not provided - synchronizer choice is left for canton participant logic.
Existing tests are updated (refactor(wallet-sdk): remove default synchronizer auto-selection #1740).
We try to show "realistic scenario" - we use multiple parties (Alice, Bob, TokenAdmin, TradingApp) we also
distribute dars to selected synchronizers (private dars to private). This contributes to relative complexity of the test code
source:
https://docs.google.com/presentation/d/1q6LpHi-wC_MO_mzf7wp5D-15j4lNeKjCMW5xmBggaTY
Example works with Token Standard V1
We have experimented with token standard V2 -> but as V2 is not yet merged to splice use of V2 means even more code
All tests run on multi-sync
This is a first PR in a series
Showcases only on-ledger part - and ensures code works.
There are 2 follow up PRs (wip)
off ledger api for tokens standard Implementation feat: multi sync example test token token standard api implementation #1782(code extracted and merged here)
Technical limitations:
In this PR we introduce separate tests for multi-sync (otherwise regular tests are flaky on multi-sync) - that split will be removed in a separate PR (above)
New example script:
docs/wallet-integration-guide/examples/scripts/15-multi-sync-trade.ts
Notes
we experimented with automatic reassignment of contracts - and it basically worked BUT.
It worked as long as
Bobis owner ofTokenRulescontract which seems unrealistic.We introduced TokenAdmin as additional party that is issuer of
Tokenonapp-synchronizer(private one).But this in fact forces us to use explicit reassignment.
But since the settlement actually happens on global - TestToken related models must be also on global synchronizer