S3 Plugin Rework Design

[ata-nas]

Goal

Store all blocks in the full Hiero Network history in large "tar" files stored in an S3-compatible "bucket" and update that bucket with newly produced blocks in near-real-time.

Intent

We plan to rework the S3 plugin to improve capability, enhance reliability, and ensure full history support. As part of this rework we also intend to continuously append blocks to "tar" archive files so that data is secured in the cloud service as quickly as possible.

Technical Notes

• This rework includes:
  • Migrate to use the "Bucky" library instead of the local client.
  • Change to use multipart upload for blocks as they arrive instead of batching them in memory and chunking them and uploading them.
  • Change to recover to within the last multipart upload segment if the node shuts down and restarts.
  • Change to determine at the last block successfully stored by the active multipart upload.
    • We need to be able to get the last block in the last successful multipart upload, which can be determined simply by reading back the last multipart item and scanning from the back (which tar supports) to read the last block in the segment. This will allow us to startup successfully after a partial upload has happened.
  • Add further support to achieve highly reliable and repeatable startup.
  • Work out a process to fill in gaps (i.e. signal backfill plugin) on startup.
  • Ensure the plugin has all necessary features to support a reliable, unbroken chain of all blocks uploaded to s3, i.e. full history.
• We need to create a formal design document that will describe the plugin in details.

NOTE: This rework will also benefit from all the rest of improvements that we need to make, like check for hashes for uploads, improve response code handling, make the client more resilient etc... With that in mind, we might choose to schedule those improvements together with, or after, this rework.