[DISCUSSION] Sharded Fungible Token

[jakmeier]

As part of #605, I am developing a reference sharded FT contract that largely works like NEP-141 tokens but using sharded contracts.

I will use this issue to document technical details that I'm finding along the way.

This is also a good place for more people to join the discussion and give feedback.

Links

High-level overview slides: https://docs.google.com/presentation/d/1GTkpeHn99W7Uh47w1CqSCMt1xHL4f8SRABw_IiFRP6w/edit?usp=sharing

Reference implementation (WIP): https://github.com/jakmeier/near-sdk-rs/tree/wip-sharded-ft/near-contract-standards/src/sharded_fungible_token

[jakmeier]

Meta data

• ft_total_supply and ft_balance_of of NEP-141 will not work on every sharded instance of the contract.
• Suggestion: ft_total_supply must work on a central account and the sharded FT implementation must ensure it is accurate.
  • The best way to do this is to only mint new tokens on the central contract.
  • Fixed supply tokens are even easier and they could also support ft_total_supply on every instance
• Suggestion: ft_balance_of(account_id: AccountId) is replaced with balance_of() that just returns the balance of the current account
  • Backwards compatibility could be done, providing ft_balance_of on the central contract, or even on all instances of the contract. But it would no longer be a view method, since it needs to make a cross-contract call.

[jakmeier]

FT Transfer

FT transfer could work in 2 blocks, thanks to local receipt execution. (Assuming we keep sharded contract instances on the same account, as currently proposed in #605. If we make them separate accounts with a different account id, this wouldn't work out-of-the-box.)

However, this only works if we know the receiver exist and they have their corresponding FT sharded contract module enabled. Otherwise, an extra block latency is required to resolve the callback that refunds the tokens to the sender.

Legend:

• blue rounded boxes: receipts
• yellow half-rounded boxes: non-technical "intention" of what's actually happening in the contract's state

[jakmeier]

FT Transfer Call

Surprisingly, we can make sharded ft transfer calls one block latency faster than the non-sharded version!

This works thanks to three things:

• the local receipts optimization in block 1
• the insight that in block 2, we don't need to go to ft@bob.near first, we can directly call the receiver contract (e.g. receiver@bob.near)
• the insight that both refund and transfer can happen in parallel in block 3
  • note: Using low-level methods of the Near Protocol wasm runtime, it is already possible to send two copies of the same output to two different callbacks. But the SDK's Promise API is not expressive enough to do that, at the time of writing. Adding support in the SDK would be simple.

Event emission

• NEP-141 always emits 2 transfer events, one in block 2 for the full amount and another in block 4 to refund the unused amount back
• The new proposal only emits one transfer event in block 3
• NEP-141 explicitly burns tokens in block 4, if alice.near is no longer a valid receiver (deleted storage reservation in ft.near). This also updates the total supply.
  • The sharded proposal doesn't really handle this case, yet. But if we don't allow deleting sharded contract instances by the user, then it cannot happen that anything needs to be burnt.
  • It's not clear if we even need to track burnt token explicitly, or if we can tolerate vanishing tokens in case of a user deleting their account during an ft_transfer_call. total_supply wouldn't be accurate in such a case, but the real total circulating supply of an FT can be different from total_supply if access to accounts is lost in other ways.

Legend:

• blue rounded boxes: receipts
• yellow half-rounded boxes: non-technical "intention" of what's actually happening in the contract's state
• orange half-rounded boxes: emitted events in logs for off-chain observers

[birchmd]

Thanks for starting this thread @jakmeier ! Having a concrete example contract is helpful when thinking through the implications of the design choices for the sharded contracts protocol feature.

I have a couple comments:

• For sft_transfer there needs to be a callback to check the success of sft_receive. You do have this in the reference implementation, but not in the diagram. I think it is important to include because it does contribute 1 block of latency from the perspective of the sender since they will be waiting for their whole transaction (including all receipts it created) to be finished.

• For sft_transfer_call, would sft_resolve_transfer be able to execute in block 2 because it is a local receipt? If so then I think it might make sense to create the refund promise inside sft_resolve_transfer instead of doing it in sft_on_transfer. The reason is because then the interface of sft_on_transfer can be exactly the same as it is for ft_on_transfer today, and I think that will simplify the lives of developers that want to migrate their contracts to handle a sharded version of a token they already handle.

[jakmeier]

@birchmd Thank you for taking a close look!

For sft_transfer there needs to be a callback to check the success of sft_receive.

100%, yes, thanks for bringing it up. I also realized that I forgot this at a random moment over the weekend after posting. This is indeed an important detail. I have now updated the comment, adding the complete diagram.

For sft_transfer_call, would sft_resolve_transfer be able to execute in block 2 because it is a local receipt?

No, at least not without further protocol changes. "Local receipts" are a special case that only work for transaction that get turned into receipts. Receipts generated from a function call are (in today's protocol) never executed in the same block.

Conceptually, it would be thinkable to make it a synchronous execution call in the future, though.

The reason is because then the interface of sft_on_transfer can be exactly the same as it is for ft_on_transfer today

I have just updated the reference implementation to return a single U128 number in sft_on_transfer.

#[serde(transparent)]
pub struct TransferCallResponseValue {
    pub unused: U128,
}

Security-wise, it is equivalent from what I had before. Sending two numbers was redundant.

The wrapper struct is transparent, so it's the same on the wire as just a U128.
Would that also address your concern regarding the interface?

[birchmd]

Yes, I like the new version of sft_transfer_call much better. The token creating all necessary promises itself simplifies the development on the receiving side compared to the original proposal, and keeps it in-line with the current NEP-141 spec. Having two promises with the same input is also a cool pattern!

[mitinarseny]

Hey guys, just left a comment on #607:

An alternative approach would be to extend design of sharded FT proposed in #614 in way, where ft1.near@alice.near can potentially store multiple separate balances intended for different "submodule" contracts within same account_id: one for "parent" alice.near, one for internet.near@alice.near, etc....

Besides that, wanted to leave couple of comments here, too:

1. Suggested above design for sft_transfer_call() allows the receiver (i.e. receiver@bob.near) to only store some data about tokens that are about to be received. It doesn't allow the receiver to further transfer just received tokens, because... they were not actually received yet: balance of ft@bob.near will be increased in ft@bob.near::sft_resolve_transfer() only after receiver@bob.near::sft_on_transfer() returns.
   This behavior is much desired by DeFi protocols to function well.

2. However, this only works if we know the receiver exist and they have their corresponding FT sharded contract module enabled. Otherwise, an extra block latency is required to resolve the callback that refunds the tokens to the sender.

   Who is responsible for initial deployment of ft@bob.near? Can alice.near send ft to bob.near if he didn't previously deploy ft@bob.near by himself? This is crucial. Otherwise, a lot of mechanics wouldn't work:

   • Airdrops
   • Sending tokens to implicit account_id, which doesn't yet have NEAR to pay for "submodule" contract deployment.
   • CEXs would need to deploy submodules for each whitelist token on passive deposit addresses by themselves, even if the user wasn't going to deposit anything.

Alternative approach

I'd like to suggest an alternative approach, which is heavily inspired by Jetton standard on TON and could look something like:

sFT Transfer

1. alice.near calls ft@alice.near::sft_transfer({"amount": "100", "receiver_id": "bob.near"}), which:
   1. Ensure that predecessor_id alice.near is allowed to make transfers
   2. Subtract amount from its own balance
   3. Call ft@bob.near::sft_receive({"amount": "100", "sender_id": "alice.near"}) along with params for implicit creation (needs to be supported by Near Protocol)
2. If ft@bob.near didn't exist yet, then it gets deployed and initialized with zero balance
3. ft@bob.near::sft_receive({"amount": "100", "sender_id": "alice.near"}):
   1. Ensure that it was called by the same version of ft global contract
   2. Add amount to its own balance
   3. Return amount

sFT Transfer Call (i.e. sft_transfer with msg)

1. alice.near calls ft@alice.near::sft_transfer({"amount": "100", "receiver_id": "bob.near", "msg": "<MESSAGE>"}), which:
   1. Ensure that predecessor_id alice.near is allowed to make transfers
   2. Subtract amount from its own balance
   3. Call ft@bob.near::sft_receive({"amount": "100", "sender_id": "alice.near", "msg": "<MESSAGE>"}) along with params for implicit creation (needs to be supported by Near Protocol)
2. If ft@bob.near didn't exist yet, then it gets deployed and initialized with zero balance
3. ft@bob.near::sft_receive({"amount": "100", "sender_id": "alice.near", "msg": "<MESSAGE>"}):
   1. Ensure that it was called by the same version of ft global contract
   2. Add amount to its own balance
   3. Call bob.near::sft_on_transfer({"amount": "100", "sender_id": "alice.near", "msg": "<MESSAGE>"}).then(ft@bob.near::sft_resolve_transfer({"amount": "100", "sender_id": "alice.near"}))
4. bob.near::sft_on_transfer({"amount": "100", "sender_id": "alice.near", "msg": "<MESSAGE>"} gets executed and, similar to NEP-141, returns amount of tokens that needs to be refunded back to sender
5. ft@bob.near::sft_resolve_transfer({"amount": "100", "sender_id": "alice.near"}) gets executed:
   1. Parse result of previous promise and set it to refund_amount (otherwise, amount in case of failure) with same checks as in NEP-141
   2. If refund_amount == 0, then return amount
   3. Call ft@alice.near::sft_receive({"amount": refund_amount, "sender_id": "bob.near", "memo": "refund"})
6. ft@alice.near::sft_receive({"amount": refund_amount, "sender_id": "bob.near", "memo": "refund"}) gets executed:
   1. Ensure that it was called by the same version of ft global contract
   2. Add refund_amount to its own balance
   3. Return refund_amount

It's less optimal in terms of number of required blocks to process fully, but handles transfer notifications in a right way.
What do you think? Will be happy to hear your feedback, so we can come up with a working solution

[jakmeier]

@mitinarseny Thanks for your inputs!

Who is responsible for initial deployment of ft@bob.near? Can alice.near send ft to bob.near if he didn't previously deploy ft@bob.near by himself? This is crucial. Otherwise, a lot of mechanics wouldn't work:

Have you seen the Implicit Subcontract Creation section in the NEP? The NEP discussion also went in the direction of needing to support airdrops and similar operations, thus we added this feature. The sharded FT design is not updated accordingly, yet.

Also note that we can never cover for a named account that does not exist. We can only create the missing subcontract on an existing account.

Other than that, I think the only difference in your alternative is about the order of depositing and the on_transfer call. I don't fully understand your goal there.

#614 (comment) above design for sft_transfer_call() allows the receiver (i.e. receiver@bob.near) to only store some data about tokens that are about to be received. It doesn't allow the receiver to further transfer just received tokens, because... they were not actually received yet: balance of ft@bob.near will be increased in ft@bob.near::sft_resolve_transfer() only after receiver@bob.near::sft_on_transfer() returns.
This behavior is much desired by DeFi protocols to function well.

In my proposal, Bob hasn't technically received the tokens but they know 100% that they will. Is the problem that they cannot start spending them in the same call? I haven't really thought about that. Good input. Can you explain to me a bit more what specific use cases require that?

But yeah, I can see that your approach solves that. At the cost of more block latencies and higher gas costs. (A function call is expensive, relatively speaking.) Maybe there is a more efficient way to solve it.

I'm also a bit wary about error cases in your alternative. In my proposal and the reference implementation, if on_transfer fails for any reason, the full amount gets refunded to Alice. This is nice since a typo in the account address, for example, will not lead to lost funds.

In your proposal, I don't know how to handle that. We can't refund on failure, since a failure could happen after Bob has already received it. I guess we have to assume Bob has received the funds and thus never refund. That means Alice will burn her funds if the account Bob does not exists.

We would also have to be very careful about contract updates. During the code update, there can be calls which see different versions of the code. Maybe devs can implement changes in a backwards compatible way. Otherwise, if a call with version mismatch simply fails, this will also lead to lost funds in your proposal, while in my proposal the FT would simply be refunded.

It's less optimal in terms of number of required blocks to process fully, but handles transfer notifications in a right way.

What do you mean by transfer notifications?

[mitinarseny]

@jakmeier Sorry, I might misunderstood Implicit Subcontract Creation section. So, it only limits NEAR amount that can be spent from "parent" account on storage staking for submodule creation, but submodules themselves can still be installed permissionlessly by default, right?

As for this one:

In my proposal, Bob hasn't technically received the tokens but they know 100% that they will. Is the problem that they cannot start spending them in the same call? I haven't really thought about that. Good input. Can you explain to me a bit more what specific use cases require that?

I've just noticed that in your suggested design receiver@bob.near::sft_on_transfer() is also irreversible when ft@bob.near::sft_resolve_transfer() fails.

As for "handling after actual receiving" use cases, example are:

• "Forwarder/proxy" contracts: receive tokens and forward it further according to inner logic. Such logic is used by an proxy-adaptor for non-18 decimals base tokens for Aurora Virtual Chains
• Escrow swaps, explained here

By transfer notification I mean an optional call to "parent" smart-contract to notify that tokens were actually received.

As for "failure" to receive:

I'm also a bit wary about error cases in your alternative. In my proposal and the reference implementation, if on_transfer fails for any reason, the full amount gets refunded to Alice. This is nice since a typo in the account address, for example, will not lead to lost funds.

I thought there is a way guarantee that sft_receive() will always get executed: we always allocate enough gas and implicitly deploy if the submodule didn't exist before. But I see now, that there might be a problem with named account ids - we cannot create them implicitly (we have no public key to add). Btw, deterministic AccountId solves that!

[jakmeier]

Thanks for clarifying a few things.

So, it only limits NEAR amount that can be spent from "parent" account on storage staking for submodule creation, but submodules themselves can still be installed permissionlessly by default, right?

Yes, permissionless installation is possible. But as you point out, the code in the limited access subcontract can't access tokens. And it can't make cross-contract calls to non-sharded contracts, either. And the caller has to pay for the storage.

Btw, #607 (comment) solves that!

Yes, but what about existing accounts using the current model? I don't think it's a complete solution if it does not work with existing accounts.

"Deterministic AccountId" is really nice concept. But maybe a bit much to pull in at the moment. The account model of NEAR is quite fundamental and I think this NEP already has quite a lot of things it changes. Introducing a new type of account would need its own NEP, which could of course be combined with features we introduce in #605.

RFQ use case

To cover a RFQ system, as you described over at #607 (comment), I think we primarily need to extend the proposed FT standard to allow holding/controlling an FT balance from another subcontract.

In theory, we don't need any extra protocol features. ft.near@alice.near right now only stores a single number, which is the total balance of FTs owned by alice. But it could store a map of balances, mapping any subcontract context to a balance. For example, if alice holds a balance of 900 and another 100 that's managed by escrow.near@alice.near:

{
  "": 900,
  "Sharded(escrow.near)": 100,
}

The smart contract code can check both the predecessor ID and the predecessor context to authorize who can access which balance. No protocol level nesting required. It all works with the features of the current proposal in #605.

I'm not sure if this extra complexity needs to be in every sharded FT contract. But it could certainly be an extension that can be supported by every FT that want to be usable in such a way.

@mitinarseny Do you think that would cover your use case?