Discussion about node sync method

[codewordneptune]

I still believe that having the node download blocks randomly can lead to a poor sync experience, and this problem is likely to become more noticeable over time, especially for new nodes with little or no history. My reasoning is the following:

1. The probability that a node randomly downloads blocks that can be applied soon to the already validated blockchain state decreases as the chain grows. For example, if the node is currently synced to block height 1000, it does not help to have blocks 1002–11000 downloaded if block 1001 is still missing, since that missing block is what would allow those 10k downloaded blocks (~10 GB) to become usable.

2. As the blockchain tip height increases, the node must probabilistically download more blocks before receiving one that can actually be applied to the current state. In other words, nodes in the future will spend more time downloading just to apply the next block than nodes do today.

3. Every downloaded block that cannot yet be applied represents work that could instead have been used to advance the synced node state and apply UTXO records, bringing the wallet balance closer to its final state.

4. The more blocks the node downloads without validating and syncing them promptly, the greater the risk that this work will be lost if the node crashes or the user restarts their computer, at least on Windows.

The change in #881 is very welcome, as it makes it possible to define the directory for downloaded blocks so they are not discarded after a computer restart, at least on Windows. That said, the default behavior is still an important part of the overall user experience. It may be worth considering whether the software should work in a convenient and intuitive way for ordinary users as well, without requiring familiarity with multiple CLI arguments.

Ideally, from UX perspective:

• Downloaded blocks are immediately or soon applied to the synced state.

With reference to discussion #827:

3. Download blocks in order. Two issues: a) Starting where? A priori you might be syncing across a reorganization and the block you think you're already synced to when sync mode starts is not canonical any more. b) Duplicate work. When you're downloading from multiple peers, you don't want to ask the peers for the same block. That's a very poor use of network resources. Moreover, the counterparty serving blocks might be in sync mode themselves and incapable of sending you specific blocks because they don't have them. In this case you want to continue by downloading what you can.

Fully agree that the node should not ask for the same blocks from peers, unless there are problems.

I'd be curious to hear what the downsides would be of using the sync method below. Note: this was drafted with AI, it is high-level, and it was generated without having the full context of Neptune Cash architecture.

---

Node sync method

Sync method that aims for three things at once:

• Apply good blocks quickly.
• Stay correct across reorgs.
• Avoid being dragged around by bad peers.

Steps

1. Sync headers first

• Collect headers from multiple peers.
• Validate header linkage and consensus rules.
• Build a fork tree and choose the branch with the most cumulative work.
• Find the last common ancestor with the local chain.

2. Download block bodies near the execution frontier

• Keep a small window of missing blocks just ahead of the last applied block.
• Assign non-overlapping ranges to peers.
• Avoid duplicate requests unless a needed block times out.

3. Validate and apply immediately

• When a block arrives, validate it.
• If its parent is already applied, execute and commit it at once.
• Then keep applying any already-buffered children in order.

4. Handle stalls gracefully

• If one missing frontier block is delayed, reassign it to another peer.
• Meanwhile continue downloading later blocks within the window.
• As soon as the gap is filled, apply buffered blocks in sequence.

5. Handle reorgs safely

• If a better branch appears, find the fork point.
• If needed, roll back to that point and replay the stronger branch.
• Keep recent undo data or snapshots to make rollback cheap.

6. Defend against malicious peers

• Never trust one peer alone for chain choice.
• Penalize invalid data heavily.
• Penalize timeouts or missing data lightly.
• Limit side-branch/orphan buffering to prevent DoS.

[codewordneptune]

Let me give real example from this morning why this is such a significant issue.

Using v0.7.0 - my node entered a sync loop at 21% yesterday. At that point, all blocks up to 6219 had been validated and synced to the local blockchain state.

I then continued syncing (downloading blocks!) from 21% to 78%, at which point I decided to close the node as a test while keeping the computer running, so the rapid-block-download folder was not deleted.

In the morning, I was able to continue successfully from 78% to 83%. Then, the one peer I was receiving blocks from got disconnected.

I ran the CLI command to unban all peers, and after that I was able to find another peer. However, the sync dropped back to 21%.
As a result, at least 16-18 GB of block data was discarded and now has to be downloaded again. None of those downloaded blocks had been used to move the validated blockchain state forwarded.

This issue has also occurred multiple times with earlier node versions, resulting in tens of gigabytes of data being downloaded and then discarded before being put to use, wasting hours of time and a significant amount of bandwidth.

I have attached the log covering the period before the peer was disconnected and after I found a new peer.

neptune_node_log.txt

[aszepieniec]

Thanks for the thought-provoking note. Some thoughts in response:

Headers-First Sync

This is how most other blockchains do it. It is a robust architecture. The reason why we don't do it are:

• Headers-first sync does not address the current bad sync experience, which comes from whatever happens after the sync-challenge-response protocol (or the headers-first sync protocol in the alternative world) approves the spawning of the sync loop.
• Our headers are a lot bigger (2KB) than other chains, leading to a lot more traffic.
• Headers-first sync does not scale. Both client and server incur a workload of O(N), which is fine if you've already established that the server is trustworthy but not as a tool to decide which servers are trustworthy and which are not.
• The sync challenge response protocol gives a good-enough-in-practice probabilistic answer to the same question that the headers-first sync answers (definitevely).
• We anticipate having succinctness, which will obviate the entire issue.

We are not opposed to having a headers-first sync. We just don't think it is the highest priority at this time. But if you have, or anyone else, has resources to spare and a desire to implement one, we will be happy to give your PR a fair consideration.

We Already Do This

3. Validate and apply immediately

4. Handle stalls gracefully

• If a better branch appears, find the fork point.
• If needed, roll back to that point and replay the stronger branch.

• Penalize invalid data heavily.
• Penalize timeouts or missing data lightly.
• Limit side-branch/orphan buffering to prevent DoS.

Towards Improving the Sync Experience

A recent PR (#881) allows the user to specify where to store and read downloaded-but-not-applied-yet blocks. By default, a temporary directory supplied by the operating system is used; and this temporary directory has a tendency to disappear across reboots or after long periods of non-use. By specifying a different, fixed directory, this PR allows the user to circumvent the problem of lost data.

(Non-)uniform block selection policy. Part of the problem of downloading blocks by height uniformly is that you are very unlikely to sample the one block you need to update the state. Perhaps we should consider modifying the distribution to something that's biased towards the sync frontier (current tip height). Mind you, there is still an argument against downloading the same successor block from 10 different peers (if you are connected to that many peers) but it might be okay to accept a negligible probability of work duplication in exchange for a better sync experience. Let me pass the question back to you. Suppose you are synced to block height X, and syncing from 10 peers who are at block height Y. How do you select which blocks to download?

[codewordneptune]

Sorry for the delay, it's been a hectic week.

Just build and ran the master after #892 was merged and wow it's nice to see the synced block height move forward. This definitely improves the sync experience, thanks!

Suppose you are synced to block height X, and syncing from 10 peers who are at block height Y. How do you select which blocks to download?

Assuming the 10 peers at block height Y are on the same chain, I'd assign each peer a distinct, consecutive range of blocks starting from the next block height the node needs. If a peer fails to deliver those blocks, then the work is assigned to another peer, and the bad peer is penalized. This would maximize the speed at which the node validates and applies blocks while minimizing the risk of downloading blocks that are later discarded before use.