M2 Delivery for Migration Ease

[ainhoa-a]

Milestone Delivery Checklist

• The milestone-delivery-template.md has been copied and updated.
• This pull request is being made by the same account as the accepted application.
• I have disclosed any and all sources of reused code in the submitted repositories and have done my due diligence to meet its license requirements.
• In case of acceptance, invoices must be submitted and payments will be transferred to the Polkadot AssetHub and/or fiat account provided in the application.
• The delivery is according to the Guidelines for Milestone Deliverables.

Link to the application pull request: w3f/Grants-Program#2437

[keeganquigley]

Thanks for the delivery @ainhoa-a I should be able to get to it by the end of the week, thanks for your patience here.

[Lederstrumpf]

Hi @ainhoa-a, nice to "meet" you again :)

Thanks for your submission - I have started looking into it.

My feedback so far doesn't strictly speaking pertain to the criteria/deliverables of M2 itself, but since - in my understanding - the app is deemed "complete" (that is: ready for the user horde) after this milestone (with M3 focused on maintenance over the next year), I'd like to address the following few issues I came across before, in my subjective ordering of descending priority:

1. The address scan currently covers derivation paths with BIP44 m/44'/coin_type'/0'/0'/X', i.e. only iterates over the address indices: https://github.com/Zondax/polkadot-web-migration/blob/d4ef045c3f4111bbc34429f752581c15fc9a333d/lib/utils/address.ts#L25.
   As a comprehensive migration assistant, it should also however check derivation paths of the format m/44'/coin_type'/Y'/0'/X', i.e. also iterate over address indices, as this is also allowed by PolkadotJS/Talisman extensions.
   For instance, the test ledger I inherited for this review had DOT under the path m/44'/354'/1'/0'/1' - these would only show in the migration assistant after I adjusted the Polkadot BIP44 path to m/44'/354'/1'/0'/0'.
   I deem this important because what's worse than not having a migration assistant is one you mistakenly trust to migrate all your funds, and then they go "missing" ;)
2. It seems the Synchronize Accounts step is currently sequential. This may not have been an issue during M1's scope, but with the variety of parachains & standalone substrate chains now added, this scanning process takes very long - even with the current conservative scanning limit of 10. This is especially an issue since ledger's will lock without user activity well before this scan completes, so either they must issue multiple rescans, or press buttons on the ledger during the syncronization frequently enough to prevent the lock (I know because I had to do that ;)).
   So I think you could improve the UX a lot by performing these synchronizations in parallel. I don't know whether retrieving the account addresses from a ledger device is inherently sequential (I assume this step can't be parallelized), but for sure the rpc queries can be, especially since they all target different endpoints. If performed in parallel, you could also afford to increase the address index scanning limit well above 10.
3. It would be nice if you could define fallback RPC endpoints in https://github.com/Zondax/polkadot-web-migration/blob/e5b492f8edfd41fd556adce62b44245a830a5acc/config/appsConfig.json - these endpoints can come and go - I already ran into this with the EdgeWare endpoint (update edgeware rpc endpoint Zondax/polkadot-web-migration#324)

[ainhoa-a]

Thanks @Lederstrumpf for the feedback. We're already working on some improvements and we'll share a new version soon.

[ainhoa-a]

Hi @Lederstrumpf we’ve published a new release that includes the following improvements:

1. To address this we have introduced a new "Deep scan" functionality allowing users to specify custom account and address index ranges for scanning Ledger devices, enabling discovery of accounts beyond default derivation paths. We have also implemented new utility functions for parsing, building, and updating BIP44 derivation paths, providing more precise control over account discovery.

2. Ledger Unlock Reminder & Optimized Sync: We are already paralelilizing the rpc calls, however, synchronization remains sequential since Ledger address generation is inherently sequential. To improve usability, we’ve added a new UI component that reminds users to keep their Ledger device unlocked during prolonged synchronization or deep scan operations.

3. To address this we have included Multi-RPC Endpoint Support: Enhanced network configuration to support multiple RPC endpoints per chain, improving connection resilience and fallback mechanisms.

https://polkadot.zondax.ch/

Hi @ainhoa-a, nice to "meet" you again :)

Thanks for your submission - I have started looking into it.

My feedback so far doesn't strictly speaking pertain to the criteria/deliverables of M2 itself, but since - in my understanding - the app is deemed "complete" (that is: ready for the user horde) after this milestone (with M3 focused on maintenance over the next year), I'd like to address the following few issues I came across before, in my subjective ordering of descending priority:

1. The address scan currently covers derivation paths with BIP44 m/44'/coin_type'/0'/0'/X', i.e. only iterates over the address indices: https://github.com/Zondax/polkadot-web-migration/blob/d4ef045c3f4111bbc34429f752581c15fc9a333d/lib/utils/address.ts#L25.
   As a comprehensive migration assistant, it should also however check derivation paths of the format m/44'/coin_type'/Y'/0'/X', i.e. also iterate over address indices, as this is also allowed by PolkadotJS/Talisman extensions.
   For instance, the test ledger I inherited for this review had DOT under the path m/44'/354'/1'/0'/1' - these would only show in the migration assistant after I adjusted the Polkadot BIP44 path to m/44'/354'/1'/0'/0'.
   I deem this important because what's worse than not having a migration assistant is one you mistakenly trust to migrate all your funds, and then they go "missing" ;)
2. It seems the Synchronize Accounts step is currently sequential. This may not have been an issue during M1's scope, but with the variety of parachains & standalone substrate chains now added, this scanning process takes very long - even with the current conservative scanning limit of 10. This is especially an issue since ledger's will lock without user activity well before this scan completes, so either they must issue multiple rescans, or press buttons on the ledger during the syncronization frequently enough to prevent the lock (I know because I had to do that ;)).
   So I think you could improve the UX a lot by performing these synchronizations in parallel. I don't know whether retrieving the account addresses from a ledger device is inherently sequential (I assume this step can't be parallelized), but for sure the rpc queries can be, especially since they all target different endpoints. If performed in parallel, you could also afford to increase the address index scanning limit well above 10.
3. It would be nice if you could define fallback RPC endpoints in https://github.com/Zondax/polkadot-web-migration/blob/e5b492f8edfd41fd556adce62b44245a830a5acc/config/appsConfig.json - these endpoints can come and go - I already ran into this with the EdgeWare endpoint (update edgeware rpc endpoint Zondax/polkadot-web-migration#324)

[Lederstrumpf]

Hi @ainhoa-a,

Thanks for your updated delivery, and apologies for my delayed response.

To address this we have introduced a new "Deep scan" functionality allowing users to specify custom account and address index ranges for scanning Ledger devices, enabling discovery of accounts beyond default derivation paths. We have also implemented new utility functions for parsing, building, and updating BIP44 derivation paths, providing more precise control over account discovery.

Thank you, this works well - has the correct level of granularity.

Ledger Unlock Reminder & Optimized Sync: We are already paralelilizing the rpc calls, however, synchronization remains sequential since Ledger address generation is inherently sequential. To improve usability, we’ve added a new UI component that reminds users to keep their Ledger device unlocked during prolonged synchronization or deep scan operations.

With parallelized rpc calls, I guess you mean that for every network being checked, you query multiple rpc endpoints in parallel?
Because it seems that the RPC queries are still sequential relative to network syncs.

Synchronization remains sequential since Ledger address generation is inherently sequential.

Ok, so querying/generating addresses from the Ledger is inherently sequential (i.e. this line), but I think the process can still be made much faster (and therefore avoid the need for the user to manually keep the device unlocked) with the following two changes:

1. Unless I misunderstood the code, the app is still processing the networks & their addresses sequentially: once a ledger address fetch is complete, it has to wait for the rpc query/queries for the current network & address. But there is no need to query the rpcs right after fetching the addresses from the ledger - one could first fetch all addresses from the ledger in one go, and thereafter perform all the rpc queries in one go.
   From a coarse benchmark on a system with a low-latency/high-bandwidth internet connection, the current process (querying the default 5 address indices per network) took ~82 seconds including the rpc queries, and only ~48 seconds if the rpc queries were skipped.
   So if one segregates the ledger interactions from the rpc queries, the ledger would have to remain unlocked manually by the user for much shorter since the ledger doesn't inherently have to remain unlocked for the rpc queries (on my example system, about 40% shorter), and the rpc queries could in fact be performed in parallel across all networks.

2. I note that you're fetching each address from the ledger, but I think this can actually be improved?
   Unless I'm misunderstanding something, the only part you really need to fetch from the ledger is the public key, and not the address: once you've fetched the public key, you can without further ledger interactions derive the corresponding address on some network from the public key and the network prefix, such as done here: https://github.com/Zondax/polkadot-web-migration/blob/b36e75bb1e1cf3e6239129f5697aac15654936bb/lib/utils/address.ts#L50-L55
   Since the code is skipping on-device address confirmation already, I don't see a security downside of doing so.
   About 40% of the networks you're supporting share coin_type (6 networks use polkadot's 354, and 3 use kusama's 434).
   So if instead of querying addresses for each network (as suggested above in 1.), you query only one network for each unique coin_type and then convert this to the other addresses, the time that the ledger has to remain unlocked over reduces by another ~40%.

So in aggregate, I estimate you'd end up with a ~65% reduction in required unlock time against the status quo if you both 1. segregate address fetching from rpc queries, and 2. only fetch the public key (per account/address index) for a given coin_type once and convert this to addresses of other networks with the same coin_type in the browser/app, rather than re-retrieving this needlessly from the ledger.

[ainhoa-a]

Hi @Lederstrumpf please find here the latest release https://github.com/Zondax/polkadot-web-migration/releases/tag/v1.4.0
https://polkadot.zondax.ch/migrate We have introduced improvements on the following areas:

• Performance and Synchronization: Introduced Ledger address caching to reduce device calls and optimized the synchronization and migration processes for improved feedback and efficiency.
• Bug Fixes: Addressed issues related to signatory recognition, deep scan destination visibility, and disabled back navigation during migration to prevent process interruptions.
• Stability and Maintenance: Updated core dependencies and refined deep scan and synchronization phases to enhance overall reliability and performance.

[Lederstrumpf]

Hi @ainhoa-a, thanks for the various updates to the PR.
Happy to accept this PR now, with the caveat of delivery of decision deposit management, as discussed on matrix.
Great work!

[github-actions]

🪙 Please fill out the invoice form in order to initiate the payment process. Please make sure that you follow the instructions and requirements as laid out in the form as well as our Terms & Conditions. Thank you!