Improve the state sync progress report

[liuchengxu]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Motivation

liuchengxu@9d98cef is collected from a warp-sync on polkadot. The percentage of 14% was reported with a list of data size ranging from 9.97MiB to 285.73 MiB (State sync, Downloading state, 14%, 9.97 Mib, State sync, Downloading state, 14%, 285.73 Mib)

Another piece of state syncing log on subcoin:

Details

2024-08-17 15:42:06.918 INFO tokio-runtime-worker substrate: ⚙️ State sync, Downloading state, 74%, 3.60 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 704.7kiB/s ⬆ 0.2kiB/s
2024-08-17 15:42:11.918 INFO tokio-runtime-worker substrate: ⚙️ State sync, Downloading state, 74%, 7.29 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 545.5kiB/s ⬆ 0.1kiB/s
2024-08-17 15:42:16.919 INFO tokio-runtime-worker substrate: ⚙️ State sync, Downloading state, 74%, 13.48 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 886.1kiB/s ⬆ 0.2kiB/s
...
2024-08-17 17:14:57.299 INFO tokio-runtime-worker substrate: ⚙️ State sync, Importing state, 0%, 3914.19 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 0.3kiB/s ⬆ 0.2kiB/s
2024-08-17 17:15:02.299 INFO tokio-runtime-worker substrate: ⚙️ State sync, Importing state, 0%, 3914.19 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 0.1kiB/s ⬆ 12 B/s
2024-08-17 17:15:07.299 INFO tokio-runtime-worker substrate: ⚙️ State sync, Importing state, 0%, 3914.19 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 92 B/s ⬆ 0
2024-08-17 17:15:12.299 INFO tokio-runtime-worker substrate: ⚙️ State sync, Importing state, 0%, 3914.19 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 92 B/s ⬆ 0
2024-08-17 17:15:17.300 INFO tokio-runtime-worker substrate: ⚙️ State sync, Importing state, 0%, 3914.19 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 0.2kiB/s ⬆ 12 B/s
2024-08-17 17:15:22.300 INFO tokio-runtime-worker substrate: ⚙️ State sync, Importing state, 0%, 3914.19 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 92 B/s ⬆ 0
2024-08-17 17:15:27.300 INFO tokio-runtime-worker substrate: ⚙️ State sync, Importing state, 0%, 3914.19 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 92 B/s ⬆ 0
2024-08-17 17:15:32.301 INFO tokio-runtime-worker substrate: ⚙️ State sync, Importing state, 0%, 3914.19 Mib (1 peers), best: #556312 (0xe1ae…c2b2), finalized #556306 (0xd31d…070c), ⬇ 0.1kiB/s ⬆ 12 B/s
2024-08-17 17:15:33.694 INFO tokio-runtime-worker sync: State sync is complete (3914 MiB), restarting block sync.

Aas shown in the logs, the progress is reported as State sync, Downloading state, 74%, 3.60 MiB, and even as the downloaded data size increases significantly up to State sync, Downloading state, 74%, 3909.93 MiB, the percentage remains fixed at 74% until the end of this phase. Similarly, during the "Importing state" phase, the progress percentage remains at 0% for the entire duration.

The current state sync progress reporting can be confusing and uninformative, especially when dealing with large state syncing.

Request

While I understand that the progress percentages may only be estimates and not perfectly precise, the current reporting is so far from accurate that it provides little value to the user.

Solution

No response

Are you willing to help with this request?

Maybe (please elaborate above)

[liuchengxu]

Since the state sync data deals with storage items, we can track the total number of storage items within the database layer and include it in the initial state response, allowing us to calculate the percentage of received items relative to the overall count. While this introduces a minor overhead, it significantly enhances the accuracy of progress reporting. I conducted a local experiment, and the results were very satisfactory—though not perfectly precise, they are good enough to provide a meaningful progress update.

The downside of this idea is the backward compatibility: a) it needs to change the structure of StateResponse, introducing the network versioning becomes crucial again. b) it mainly benefits new nodes, as older nodes lack awareness of the total storage item count in the state. We can retain the existing progress report for older nodes. If needed, we could also consider adding a method to initialize this count for older nodes.

cc @bkchr @dmitry-markin

[dmitry-markin]

All the protocol changes should go through RFCs. I am unsure if it is feasible to change the protocol for improved metering only, but if grouped with some important state sync protocol upgrade this can be done more easily.

[liuchengxu]

That's fair, just want some early discussion before opening a new RFC. There are already an RFC blocked by the network protocol versioning support, see this discussion polkadot-fellows/RFCs#112 (comment). I think the feature request here is doable once we have some sort of network protocol versioning, but I doubt the versioning addition is better to be initiated by the external contributors.

[liuchengxu]

For anyone interested in this issue, here's another approach that doesn't require modifying Substrate. You can introduce a custom request-response protocol to query the total number of key-value pairs in the state (of course, you should track this number first in your node). Then, add an additional progress indicator based on the ratio of received key-value pairs to the total. This method has already been implemented in Subcoin and works well in practice.

[bkchr]

Yeah I think in general there is probably nothing bad in improving this. Feel free to open a RFC.