Description
Story Form
I would like archive block backups to be started as soon as possible to minimize risk of loosing block data.
Technical Notes
The first implementation waits till all blocks are available before starting uploading a backup archive file. So for example if the backup size is set at 10,000 blocks, the archive plugin waits till the next 10,000 blocks are available and then starts uploading them into new "tar" file. This means that if a disaster happens while waiting for 10,000 blocks then up to 10,000 blocks could be lost forever. The archive plugin uses S3 multi-part uploads, there is no time limit to complete upload. So we could start uploading as soon as there is a 5MB chunk of block data to upload. We will need to carefully store metadata of the multi-part key and the last block written and offset in that block. That way if block node is restarted in the middle of upload it can restart where it left off. In a disaster recovery case you will be able to manually close the multi-part upload and download its contents. It will have recoverable contents or all blocks other than last 5MB of data. So worst case scenario you only loose 5Mb of block data.