Skip to content

Add support for ProgressBars to eta.core.storage #477

@brimoor

Description

@brimoor

It would be nice to have the option to render a progress bar when uploading/downloading files via the eta.core.storage module, to provide a better experience when transferring large files.

For HTTPSStorageClient, we use requests, so it should be easy enough to support progress bars via the approach outlined in:

eta/eta/core/web.py

Lines 136 to 146 in 03d3101

r = self._get_streaming_response(url, params=params)
etau.ensure_basedir(path)
size_bytes = _get_content_length(r)
size_bits = 8 * size_bytes if size_bytes is not None else None
with etau.ProgressBar(size_bits, use_bits=True) as progress:
with open(path, "wb") as f:
for chunk in r.iter_content(chunk_size=self.chunk_size):
f.write(chunk)
progress.update(8 * len(chunk))

boto3 seems to support callbacks for upload/download functions, which should make this pretty easy for AWSStorageClient:
https://stackoverflow.com/q/41827963

Google Cloud doesn't seem to (yet?) support callbacks, so GoogleCloudStorageClient will take a bit more work:
googleapis/google-cloud-python#15992

Metadata

Metadata

Assignees

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions