download to s3 #1325
Description
As a user I would want a download plugin that directly forwards HTTP response to a given S3 bucket without using local file system.
Chunk size, target url, source s3 bucket and key, credentials should be configurable.
Example using boto3:
with requests.get(stream_url, stream=True, auth=auth) as response:
response.raise_for_status() # Raise an error for bad responses (4xx and 5xx)
chunk_size = 64 * 1024 # 64kb
with response.raw as data_stream:
boto3_client.upload_fileobj(data_stream, bucket, key, Config=boto3.s3.transfer.TransferConfig(multipart_threshold=chunk_size * 2))