Tracking issue for XCM Revamp

[gavofyork]

Follow-up issue: paritytech/polkadot-sdk#925

Priorities

• Harden the execute_xcm function against recursion-bombs (Guard against XCM recursive bombs by setting a recursion limit #3555, Harden XCM v1 for Recursions #3586)
• Type-safe wildcard management in MultiAsset (Typesafe MultiAsset Wildcards  #3168).
• Implement proper weight pricing for the queuing and dispatch - it should integrate into the chain's usual transaction weight pricing. (Add XCM Origin and converter #2896)
• Don't just burn fees - send them somewhere, and test that we get them! (Add XCM Origin and converter #2896)
• Avoid using burn & mint in teleport - instead transfer into and out of a special reserved "teleported" account. (Check out/in assets when teleporting to maintain total issuance #3007)
• Introduce a reserve_asset_transfer extrinsic and execution filter enabling it (Introduce reserve-asset transfer extrinsic #3010)
• Add a register_id function to paras_registrar (Extrinsic for reserving a parachain ID #3008)
• Typesafe MultiAsset Wildcards  #3168: Typesafe MultiAsset wildcards. (XCM v1 #2815)
• DepositAsset and DepositReserveAsset should have an extra parameter limiting the number of distinct assets to deposit. #3547: DepositAsset and DepositReserveAsset should have an extra parameter limiting the number of distinct assets to deposit. (XCM v1 #2815)
• Introduce internal Vec<Order> field for BuyExecution. (XCM v1 #2815)
• Update XCM format document to reflect new format (Update xcm-format document to reflect new format. polkadot-fellows/xcm-format#23).

Substantial features

• General ResponseMap of (QueryId, Origin) -> EitherOr<Dispatch, Option<Response>> which allows free execution of the message and then either stores the result or calls a dispatch - in this case it could go overweight be up to the weight of whatever dispatch is in there. It should be added on and checked before executing any further XCMs. (XCM v2: Scripting, Query responses, Exception handling and Error reporting #3629)
• Prepaid outcome reporting mechanism for success/failure. (XCM v2: Scripting, Query responses, Exception handling and Error reporting #3629)
• New instruction RefundSurplusWeight which trades surplius weight in and places it back in holding. (XCM v2: Scripting, Query responses, Exception handling and Error reporting #3629)
• SetErrorHandler instruction with pre-paid execution of XCM in case of error. (XCM v2: Scripting, Query responses, Exception handling and Error reporting #3629)
• In the case of an incompletely executed XCM, return the weight consumed, not the maximum. (XCM v2: Scripting, Query responses, Exception handling and Error reporting #3629)
• AssetTrap instruction which places any remaining assets in the holding account into a special storage item (keyyed under the origin location), even (especially) in the case of error. A secondary XCM instruction DrainTrap can fetch those assets and do something with them. A failsafe asset trap could also be in place for the origin. XCM: Asset trap to avoid accidental burning of Holding Register contents #3694 (XCM: Allow reclaim of assets dropped from holding #3727)
• XCM Pallet should implement WrapVersion. Introduce QueryVersion and NoteVersion message types, allowing dynamic negotiation of transport version, followed with persistent transport version storage for each origin. (XCM: Automatic Version Negotiation #3736)

VMP Origins

Figure out how to manage origin permissions. We want root and governance Relay-chain XCM origins to have free execution on the Statemint parachain, but not just random accounts that happen to be instructing the Relay-chain to send an XCM on their behalf to the parachain.

• Introduce RelayedFrom XCM (XCM Revamp Continued #2865)
• Introduce RelayedFrom into execute_xcm, checking it's a subconsensus, mutate origin, redispatch; ensure that mutating the origin doesn't create problems for relative locations in e.g. asset descriptors. (XCM Revamp Continued #2865)
• Introduce a means of converting a Collective (pallet) Origin into a Plurality MultiLocation - this will likely be through a Origin -> MultiLocation conversion utility trait. (XCM Revamp Continued #2865)
• Introduce XCM sender dispatchable using RelayedFrom XCM: (XCM Revamp Continued #2865)
  • Introduce an XCM dispatcher into Polkadot, equivalent of xcm-handler in Cumulus, it should allow for RelayedFrom messages to be created and sent (search fn send(origin, dest: MultiLocation).
  • Use pallet-xcm in Cumulus and remove duplicate functionality from xcm-handler.
  • Will use the new Origin -> MultiLocation converter. (XCM Revamp Continued #2865)
  • With the MultiLocation it should potentially (if non-Null) wrap the XCM in a Xcm::RelayedFrom. (XCM Revamp Continued #2865)
  • Should be configurable to have a barrier (possibly by just using existing Origin -> MultiLocation conversion) (XCM Revamp Continued #2865)
• Introduce XCM Handler pallet a Origin::Xcm(MultiLocation)
• Introduce ensure_xcm and EnsureXcm impl EnsureOrigin origin barrier and use EnsureXcm([Parent, Plurality(Executive, >1/2)]) for Admin in Statemint.
• Provide Origin converter to use the Origin::Xcm Add XCM Origin and converter #2896
• Check that AccountId origins from Relay have sensible origins on Paras when sent over DMP.

Queuing

XCMP queueing mechanisms should be largely replicated for VMP queuing.

• UMP outbound queuing.
• UMP inbound queuing & weight limits, similar to DMP (immediate processing, overweight message should not block queue, don't stop processing when one para blocked) (UMP: Support Overweight messages #3575).
• DMP outbound queuing.
• DMP incoming queue & weight limits. (DMP Queue pallet cumulus#416)

Smaller features

• Implement transfer_asset for FungiblesAdapter, but locate the function in a different trait to avoid the need to implement only when deposit+withdraw are implemented. (XCM Revamp Continued #2865)

Minor refactorings/additions/optimisations

• More tests in xcm-executor/xcm-builder for new logic, especially for Transact execution origins (XCM docs and tests #2948).
• Merge Filter and Contains. (Remove Filter and use Contains instead substrate#9514)
• Tests for everything in xcm-builder (XCM docs and tests #2948, add integration tests to xcm-builder #3537).
• Switch to using new MultiAsset. (XCM v1 #2815)
• Update Junction when Support compact encoding for tuple enum items parity-scale-codec#262 is done.
• LocationConversion should be removed in favour of using the Convert trait. (XCM revamp #2836)
• Additional two dispatchables for fn authorize_upgrade(origin: Root, code_hash: T::Hash) and fn enact_authorized_upgrade(origin: Any, code: Vec<u8>) (Two-stage upgrade for parachains cumulus#392)
• Introduce Junctions separately to MultiLocation (Ensure MultiLocation always has a canonical representation #3404)

XCM v3:

• BurnAsset instruction.
• ExpectAsset, ExpectOrigin, Expect... instructions.
• Safe version of the Transact instruction, allowing version-sensitive dispatching between chains. Discoverability and versioning for transacting polkadot-fellows/xcm-format#22 (#

Things to consider:

• Message prioritisation/QoS. Currently all messages in a channel must be executed in strict order of being sent. If the receiving system runs out of weight while processing due to a heavy message, then other channels will be tried, even if there are lighter messsages following it which could reasonably be executed. For now this is fine because a) we have an optimised data system for taking only contiguous messages from the beginning of the page stored in state and b) during execution it gives other channels with lighter messages a chance to execute.
• Scheduling. Right now we process all channels' messages, one page at a time in a random-shuffled round-robin, incrementally making more weight available until we have made all weight available and run out of weight to process the next message on every channel. This works and is safe, but might lead to slightly unexpected stuff happening such as mesages sent much later on a lightly-used channel executing before messages sent on a heavier-used channel.