See PR #1162 for initial conception.
The role of the bootstrap algorithm is to start the Celestia client to ensure that it doesn't miss any block digests and doesn't send the same digest twice. The Celestia client buffers block digests in memory before sending them in a blob. If a crash occurs, these digests are lost. The role of the bootstrap is to recover these digest and the new one created during the crash. The celestial client can fail an the main node process still producing block.
Since all digests sent to the Celestia client are stored by the storage, they can be requested. The goal of this algorithm is to determine which digests should be requested using the last sent digest to Celestia and the last Celestia DA (Data Availability) blob processed.
The algorithm can use two types of trust sources:
- The Celestia network: Be cautious when the node starts, as some blobs may still be produced by the Celestia network. The function needs to wait to ensure that no blob is pending in the Celestia network, then fetch the last finalized blob to examine the last processed blob digest and obtain the corresponding block to determine the last push block height.
- The local database: The blocks sent and the finalized block heights (both DA and block) are saved in the database and can be used for determining the bootstrap height. However, if some digests in the database are marked as sent but not finalized, they must be verified against the Celestia network to see if they have been finalized during the node crash, or the algorithm must wait for their processing to complete.
When the next block digest height is determined, the algorithm fetches all blocks up to the current last block height and sends them to Celestia in one or several blobs (if the maximum blob size is reached), containing all pending digests.
Once this is done, the new block digests added during the bootstrap process are integrated into the normal Celestia process.
The normal Celestia process aggregates all received block digests and sends them in blobs when the Celestia network is available and no blob is being processed.
This PR concentrates on the main bootstrap algo and not all Celestia stuff. If some Celestia data are needed they will be simulated via a specific mock function.
To query the block storage, the Celestia client has a specific channel to send request. It's an asynchrone process that use oneshot channel to get the answer-back. In the current impl the oneshot channel has to be added. See this example.
This process is not currently connected but it should be in next PRs. The connection is not part of this PR. To test the channel can be used to simulate thee storage.
A significant part of the development involves creating extensive tests that cover all possible cases. Since the integrity of the chain on Celestia depends mostly on this algorithm, particular attention should be given to testing to ensure all possibilities are covered.
See PR #1162 for initial conception.
The role of the bootstrap algorithm is to start the Celestia client to ensure that it doesn't miss any block digests and doesn't send the same digest twice. The Celestia client buffers block digests in memory before sending them in a blob. If a crash occurs, these digests are lost. The role of the bootstrap is to recover these digest and the new one created during the crash. The celestial client can fail an the main node process still producing block.
Since all digests sent to the Celestia client are stored by the storage, they can be requested. The goal of this algorithm is to determine which digests should be requested using the last sent digest to Celestia and the last Celestia DA (Data Availability) blob processed.
The algorithm can use two types of trust sources:
When the next block digest height is determined, the algorithm fetches all blocks up to the current last block height and sends them to Celestia in one or several blobs (if the maximum blob size is reached), containing all pending digests.
Once this is done, the new block digests added during the bootstrap process are integrated into the normal Celestia process.
The normal Celestia process aggregates all received block digests and sends them in blobs when the Celestia network is available and no blob is being processed.
This PR concentrates on the main bootstrap algo and not all Celestia stuff. If some Celestia data are needed they will be simulated via a specific mock function.
To query the block storage, the Celestia client has a specific channel to send request. It's an asynchrone process that use oneshot channel to get the answer-back. In the current impl the oneshot channel has to be added. See this example.
This process is not currently connected but it should be in next PRs. The connection is not part of this PR. To test the channel can be used to simulate thee storage.
A significant part of the development involves creating extensive tests that cover all possible cases. Since the integrity of the chain on Celestia depends mostly on this algorithm, particular attention should be given to testing to ensure all possibilities are covered.