-
Notifications
You must be signed in to change notification settings - Fork 3
Open
Labels
Description
AWS Multipart upload seems to only be allowed if the file to upload is ≥5MB.
Context
- 5MB is indeed the minimum size for each part of a multipart-upload.
- That minimum part size doesn't apply to the last part of a multipart upload (which makes sense, because the total file size is rarely an entire multiple of the part size, so there's always some remainder), so one could have thought that it would still be valid to use Multipart Upload for a <5MB file, by making it a multipart operation… with a single part (it thus being the last part and not subject to the minimum)
- But it seems that this is not the case, as we encountered an error when trying to upload a <5MB git cache archive in the past — see Only publish git mirror if the archive file is 50 MB or larger #72
Action Plan
- Double-check if the error we encountered during Only publish git mirror if the archive file is 50 MB or larger #72 was indeed due to AWS not allowing Multipart Uploads for files <5MB (aka creating a multipart upload with only a single part), as opposed to a separate quirk in our implementation
- I doubt it, but since that would lead to a simple fix and avoid implementing extra stuff, worth checking anyway)
- If AWS indeed doesn't allow single-part MultipartUpload operations:
- Implement direct (aka non-Multipart) upload operation
- Implement the logic in tinys3 to choose the right operation (Multipart vs Direct) when asked to upload a file
- Note: the cutoff for the decision could (?) be greater than 5MB 🤔 maybe we don't need to use multipart-upload for files under 10MB? 🤷