Interface to expose `Call` enum structure from `construct_runtime` for XCM `Transact` callers

[KiChjang]

Right now, the way to construct a Transact XCM safely is to import the foreign chain's runtime crate and use its Call enum to encode/decode incoming/outgoing XCM Transact instructions. This is obviously not a practical or scalable solution, since a runtime crate is pretty large in size, and pulling it in as a dependency is going to considerably bloat code sizes, not to mention elongate compilation times.

What teams do nowadays instead is to have a local representation of the SCALE-encoded foreign Call enum. This is not ideal either since it can easily go out of sync with the foreign chain, which would make Transact instructions to the foreign fail.

We'll need to create some kind of interface based on the Call enum without requiring devs to pull in an entire runtime crate.

[bkchr]

Isn't this what subxt is for? It generates you a Call enum from the metadata?

[xlc]

Duplicated to the request I made more than a year ago: paritytech/substrate#8158

[xlc]

And maybe the proper solutions is polkadot-fellows/xcm-format#22 + PSPs?

[coax1d]

Duplicated to the request I made more than a year ago: paritytech/substrate#8158

The example constructed for the xcm simulator in paritytech/polkadot#5598 regarding the origin of this issue was inspired by the Acala team shown here:
https://github.com/AcalaNetwork/Acala/blob/5a8f381a95649eb74385fb7db31421d78118a651/modules/relaychain/src/lib.rs#L39-L100

[KiChjang]

Okay so I gave this some more thought, and the kinds of data that we really need is the following:

1. Pallet index
2. Call index
3. Argument types

The pallet index is generated by construct_runtime, whereas the rest is generated by the #[frame_support::pallet] attribute macro.

What this then means is that we can simply modify the two macros and have them generate some sort of interface, and put these data into a new crate. Foreign runtimes can then simply import this newly auto-generated crate. Ideally, this crate will be generated and updated whenever there's a new release -- this would then allow foreign runtimes to stay up-to-date with the latest pallet indices of the XCM Transact recipient's runtime.

[KiChjang]

I'm still curious as to whether scale-info or frame-metadata already has tools or functions that make the process easier...

[bkchr]

I will repeat myself again: https://docs.rs/subxt-codegen/latest/subxt_codegen/

[coax1d]

Just to note..

I have used in a test subxt to pull the runtime meta data which I am interested in and actually use that Runtime to decode a particular xcm Transact message.

[KiChjang]

It would be much better if the runtime can simply produce artifacts of its own interface as part of its building process. Using an external tool like subxt to extract runtime metadata is pretty brittle and can quickly get out of date as it does not keep track of runtime upgrades.

What seems to be the best solution to me is the ability for substrate-based chains to query other chains for their RuntimeCall structure, and have this metadata stored on-chain along with its runtime version, similar to how we do version negotiation in the XCM pallet.

[coax1d]

I do think that is a more robust solution when I look at it at face value.

[bkchr]

What seems to be the best solution to me is the ability for substrate-based chains to query other chains for their RuntimeCall structure, and have this metadata stored on-chain along with its runtime version, similar to how we do version negotiation in the XCM pallet.

Why should you store the RuntimeCall of another chain in your state? What does this brings you?

[KiChjang]

I'm not sure why this question is asked; it's akin to asking "why should we store the XCM version of another chain in the local chain state". We currently do so for versions because we have to figure out what the appropriate XCM version that we want to use; similarly, we should also then store a compressed version of another chain's RuntimeCall locally in order to know how to use Transact correctly.

Frankly speaking, we don't necessarily need to store the RuntimeCall -- all we need are the 3 pieces of data that I mentioned above in order to use Transact correctly (i.e. pallet indices, call indices and argument types).

[bkchr]

"why should we store the XCM version of another chain in the local chain state"

There is a huge difference. XCM needs to know the supported XCM version of the remote chain to send a XCM message in the correct format. For transact, XCM doesn't care on what the destination supports expects. It will not interpret or try to "correct" the transact message.

The transact message is constructed in the upper layers, in the chain logic. Why should XCM even try to care what kind of bytes you want to send in transact? Maybe you are speaking with a none Substrate based chain that is expecting quite a different format for the message?

If the runtime call format of the target chain changed, what would you do? A call that your pallet wants to execute requires a new argument? What will you do? Your pallet can not handle this 🤷

[KiChjang]

We obviously will key on the runtime version of the foreign runtime. Then it would simply check whether the versions match, and if not, we fail the transaction. It doesn't need to try and upgrade or downgrade your transaction.

The only thing this is really protecting against is the change in pallet indices; for the other 2 pieces of data, we can already use ExpectPallet or QueryPallet in XCMv3 to first inquire about the expected version of the pallet. As such, we don't even need to store any of these data either; just the runtime version is enough.