It would be nice to have the option to render a progress bar when uploading/downloading files via the eta.core.storage module, to provide a better experience when transferring large files.
For HTTPSStorageClient, we use requests, so it should be easy enough to support progress bars via the approach outlined in:
|
r = self._get_streaming_response(url, params=params) |
|
etau.ensure_basedir(path) |
|
|
|
size_bytes = _get_content_length(r) |
|
size_bits = 8 * size_bytes if size_bytes is not None else None |
|
|
|
with etau.ProgressBar(size_bits, use_bits=True) as progress: |
|
with open(path, "wb") as f: |
|
for chunk in r.iter_content(chunk_size=self.chunk_size): |
|
f.write(chunk) |
|
progress.update(8 * len(chunk)) |
boto3 seems to support callbacks for upload/download functions, which should make this pretty easy for AWSStorageClient:
https://stackoverflow.com/q/41827963
Google Cloud doesn't seem to (yet?) support callbacks, so GoogleCloudStorageClient will take a bit more work:
googleapis/google-cloud-python#15992
It would be nice to have the option to render a progress bar when uploading/downloading files via the
eta.core.storagemodule, to provide a better experience when transferring large files.For
HTTPSStorageClient, we userequests, so it should be easy enough to support progress bars via the approach outlined in:eta/eta/core/web.py
Lines 136 to 146 in 03d3101
boto3seems to support callbacks for upload/download functions, which should make this pretty easy forAWSStorageClient:https://stackoverflow.com/q/41827963
Google Cloud doesn't seem to (yet?) support callbacks, so
GoogleCloudStorageClientwill take a bit more work:googleapis/google-cloud-python#15992